2023-11-24T06:06:31.569 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-24T06:06:31.569 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556 --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} -- /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/orig.config.yaml 2023-11-24T06:06:31.626 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-24T06:06:31.714 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556 branch: rishabh-mgr-vol-misc description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} email: null first_in_suite: false job_id: '7465556' kernel: client: sha1: distro flavor: default kdb: true sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: rishabh-mgr-vol-misc ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-mgr-vol-misc sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3305 sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-mgr-vol-misc suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestVolumes - tasks.cephfs.test_volumes.TestSubvolumeGroups - tasks.cephfs.test_volumes.TestSubvolumes - tasks.cephfs.test_subvolume.TestSubvolume teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 2442239d2653456406c25ae0c71b689c8f2657b6 timestamp: 2023-11-24_05:41:33 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-24T06:06:31.714 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa; will attempt to use it 2023-11-24T06:06:31.715 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks 2023-11-24T06:06:31.715 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-24T06:06:31.717 INFO:teuthology.task.internal:Checking packages... 2023-11-24T06:06:31.741 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'a13ea5ff00cb4df7a9cc941166ddeb358881144c' 2023-11-24T06:06:31.741 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-24T06:06:31.742 INFO:teuthology.packaging:ref: None 2023-11-24T06:06:31.742 INFO:teuthology.packaging:tag: None 2023-11-24T06:06:31.742 INFO:teuthology.packaging:branch: rishabh-mgr-vol-misc 2023-11-24T06:06:31.742 INFO:teuthology.packaging:sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:06:31.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rishabh-mgr-vol-misc 2023-11-24T06:06:31.891 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7484.ga13ea5ff 2023-11-24T06:06:31.891 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-24T06:06:31.896 INFO:teuthology.task.internal:no buildpackages task found 2023-11-24T06:06:31.896 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-24T06:06:31.932 INFO:teuthology.task.internal:Saving configuration 2023-11-24T06:06:31.949 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-24T06:06:31.956 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T06:06:31.979 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 06:01:31.131005', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-24T06:06:32.004 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 06:01:31.132501', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-24T06:06:32.005 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-24T06:06:32.010 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-11-24T06:06:32.010 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-11-24T06:06:32.010 INFO:teuthology.run_tasks:Running task console_log... 2023-11-24T06:06:32.103 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2d7f213b80>, signals=[15]) 2023-11-24T06:06:32.104 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-24T06:06:32.109 INFO:teuthology.task.internal:Opening connections... 2023-11-24T06:06:32.109 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:06:32.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T06:06:32.178 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:06:32.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T06:06:32.244 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-24T06:06:32.250 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2023-11-24T06:06:32.270 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2023-11-24T06:06:32.271 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2023-11-24T06:06:32.329 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Red Hat Enterprise Linux" 2023-11-24T06:06:32.329 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="8.6 (Ootpa)" 2023-11-24T06:06:32.329 INFO:teuthology.orchestra.run.smithi089.stdout:ID="rhel" 2023-11-24T06:06:32.330 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="fedora" 2023-11-24T06:06:32.330 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="8.6" 2023-11-24T06:06:32.330 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el8" 2023-11-24T06:06:32.330 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-24T06:06:32.330 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2023-11-24T06:06:32.331 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-24T06:06:32.331 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.redhat.com/" 2023-11-24T06:06:32.331 INFO:teuthology.orchestra.run.smithi089.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-24T06:06:32.331 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-24T06:06:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:06:32.332 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-24T06:06:32.332 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-24T06:06:32.332 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-24T06:06:32.332 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-24T06:06:32.333 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2023-11-24T06:06:32.440 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2023-11-24T06:06:32.460 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2023-11-24T06:06:32.460 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2023-11-24T06:06:32.519 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Red Hat Enterprise Linux" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8.6 (Ootpa)" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:ID="rhel" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="fedora" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8.6" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-24T06:06:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.redhat.com/" 2023-11-24T06:06:32.521 INFO:teuthology.orchestra.run.smithi123.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-24T06:06:32.521 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-24T06:06:32.521 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:06:32.521 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-24T06:06:32.521 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-24T06:06:32.521 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-24T06:06:32.521 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-24T06:06:32.521 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2023-11-24T06:06:32.545 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-24T06:06:32.555 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-24T06:06:32.616 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-24T06:06:32.616 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2023-11-24T06:06:32.619 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2023-11-24T06:06:32.638 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-24T06:06:32.644 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-24T06:06:32.644 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2023-11-24T06:06:32.678 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2023-11-24T06:06:32.704 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-24T06:06:32.757 INFO:teuthology.run_tasks:Running task kernel... 2023-11-24T06:06:32.771 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'} 2023-11-24T06:06:32.771 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-11-24T06:06:32.771 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-11-24T06:06:32.772 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-24T06:06:32.772 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-24T06:06:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:06:32.790 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2023-11-24T06:06:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:06:32.791 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-11-24T06:06:32.849 INFO:teuthology.orchestra.run.smithi089.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-24T06:06:32.849 INFO:teuthology.task.kernel:Running kernel on smithi089: 4.18.0-372.9.1.el8.x86_64 2023-11-24T06:06:32.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2023-11-24T06:06:32.850 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-24T06:06:32.851 INFO:teuthology.task.kernel:Running kernel on smithi123: 4.18.0-372.9.1.el8.x86_64 2023-11-24T06:06:32.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2023-11-24T06:06:33.995 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-11-24T06:06:33.995 INFO:teuthology.orchestra.run.smithi089.stdout:Unable to read consumer identity 2023-11-24T06:06:34.031 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2023-11-24T06:06:34.031 INFO:teuthology.orchestra.run.smithi123.stdout:Unable to read consumer identity 2023-11-24T06:06:35.556 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2023-11-24T06:06:35.558 INFO:teuthology.orchestra.run.smithi089.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2023-11-24T06:06:36.940 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-11-24T06:06:37.800 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 16 MB 00:02 2023-11-24T06:06:40.456 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 484 kB/s | 24 kB 00:00 2023-11-24T06:06:41.159 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-24T06:06:41.219 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-24T06:06:41.220 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-24T06:06:41.220 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-24T06:06:41.258 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 570 kB/s | 24 kB 00:00 2023-11-24T06:06:41.357 DEBUG:teuthology.orchestra.run.smithi123:> echo no | sudo yum reinstall kernel || true 2023-11-24T06:06:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2023-11-24T06:06:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:Unable to read consumer identity 2023-11-24T06:06:41.927 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-24T06:06:41.951 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:01 ago on Fri 24 Nov 2023 06:06:40 AM UTC. 2023-11-24T06:06:41.985 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-11-24T06:06:41.986 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-11-24T06:06:41.986 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-11-24T06:06:41.992 INFO:teuthology.orchestra.run.smithi123.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-24T06:06:42.019 INFO:teuthology.orchestra.run.smithi123.stderr:Error: No packages marked for reinstall. 2023-11-24T06:06:42.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum reinstall -y kernel || true 2023-11-24T06:06:42.122 DEBUG:teuthology.orchestra.run.smithi089:> echo no | sudo yum reinstall kernel || true 2023-11-24T06:06:42.393 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2023-11-24T06:06:42.394 INFO:teuthology.orchestra.run.smithi123.stdout:Unable to read consumer identity 2023-11-24T06:06:42.494 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-11-24T06:06:42.494 INFO:teuthology.orchestra.run.smithi089.stdout:Unable to read consumer identity 2023-11-24T06:06:42.652 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:02 ago on Fri 24 Nov 2023 06:06:40 AM UTC. 2023-11-24T06:06:42.690 INFO:teuthology.orchestra.run.smithi123.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-24T06:06:42.715 INFO:teuthology.orchestra.run.smithi123.stderr:Error: No packages marked for reinstall. 2023-11-24T06:06:42.758 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:01 ago on Fri 24 Nov 2023 06:06:41 AM UTC. 2023-11-24T06:06:42.761 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-11-24T06:06:42.800 INFO:teuthology.orchestra.run.smithi089.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-24T06:06:42.833 INFO:teuthology.orchestra.run.smithi089.stderr:Error: No packages marked for reinstall. 2023-11-24T06:06:42.839 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-24T06:06:42.841 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-24T06:06:42.841 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-24T06:06:42.841 DEBUG:teuthology.parallel:result is None 2023-11-24T06:06:42.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum reinstall -y kernel || true 2023-11-24T06:06:43.206 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-11-24T06:06:43.206 INFO:teuthology.orchestra.run.smithi089.stdout:Unable to read consumer identity 2023-11-24T06:06:43.465 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:02 ago on Fri 24 Nov 2023 06:06:41 AM UTC. 2023-11-24T06:06:43.506 INFO:teuthology.orchestra.run.smithi089.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-24T06:06:43.531 INFO:teuthology.orchestra.run.smithi089.stderr:Error: No packages marked for reinstall. 2023-11-24T06:06:43.582 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2023-11-24T06:06:43.662 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-24T06:06:43.662 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-24T06:06:43.662 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-24T06:06:43.662 DEBUG:teuthology.parallel:result is None 2023-11-24T06:06:43.663 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-24T06:06:43.672 INFO:teuthology.task.internal:Creating test directory... 2023-11-24T06:06:43.672 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-24T06:06:43.675 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-24T06:06:43.694 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-24T06:06:43.725 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-24T06:06:43.757 INFO:teuthology.task.internal:Creating archive directory... 2023-11-24T06:06:43.757 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-24T06:06:43.759 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-24T06:06:43.786 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-24T06:06:43.793 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-24T06:06:43.793 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-24T06:06:43.817 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-24T06:06:43.854 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T06:06:43.859 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T06:06:43.872 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T06:06:43.879 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T06:06:43.880 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-24T06:06:43.889 INFO:teuthology.task.internal:Configuring sudo... 2023-11-24T06:06:43.889 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-24T06:06:43.923 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-24T06:06:43.958 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-24T06:06:43.967 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-24T06:06:43.967 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-24T06:06:43.991 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-24T06:06:44.018 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T06:06:44.089 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T06:06:44.146 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T06:06:44.230 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T06:06:44.273 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:06:44.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-24T06:06:44.350 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T06:06:44.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T06:06:44.450 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T06:06:44.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T06:06:44.579 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:06:44.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-24T06:06:44.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2023-11-24T06:06:44.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2023-11-24T06:06:44.705 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T06:06:44.748 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T06:06:45.052 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-24T06:06:45.060 INFO:teuthology.task.internal:Starting timer... 2023-11-24T06:06:45.060 INFO:teuthology.run_tasks:Running task pcp... 2023-11-24T06:06:45.103 INFO:teuthology.run_tasks:Running task selinux... 2023-11-24T06:06:45.142 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-24T06:06:45.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2023-11-24T06:06:45.194 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2023-11-24T06:06:45.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2023-11-24T06:06:45.250 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2023-11-24T06:06:45.252 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-24T06:06:45.252 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2023-11-24T06:06:45.278 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2023-11-24T06:06:45.278 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2023-11-24T06:06:45.303 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2023-11-24T06:06:45.303 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive'} 2023-11-24T06:06:45.304 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T06:06:45.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:06:45.344 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T06:06:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:06:45.371 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-24T06:06:45.371 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2023-11-24T06:06:45.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2023-11-24T06:06:45.440 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-24T06:06:45.449 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-24T06:06:45.454 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-24T06:06:45.574 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-24T06:06:45.590 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-24T06:06:45.591 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-24T06:15:28.369 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com')] 2023-11-24T06:15:28.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-11-24T06:15:28.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T06:15:28.440 DEBUG:teuthology.orchestra.run.smithi089:> true 2023-11-24T06:15:28.508 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-11-24T06:15:28.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-11-24T06:15:28.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T06:15:28.577 DEBUG:teuthology.orchestra.run.smithi123:> true 2023-11-24T06:15:28.644 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-11-24T06:15:28.644 INFO:teuthology.run_tasks:Running task clock... 2023-11-24T06:15:28.655 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-24T06:15:28.656 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-24T06:15:28.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T06:15:28.658 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-24T06:15:28.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T06:15:28.689 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-24T06:15:28.704 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-24T06:15:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-24T06:15:28.733 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-24T06:15:28.790 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2023-11-24T06:15:28.802 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2023-11-24T06:15:28.812 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2023-11-24T06:15:28.816 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-24T06:15:28.824 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2023-11-24T06:15:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-24T06:15:28.839 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-24T06:15:28.853 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-24T06:15:28.918 INFO:teuthology.orchestra.run.smithi089.stderr:bash: ntpq: command not found 2023-11-24T06:15:28.922 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T06:15:28.922 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2023-11-24T06:15:28.922 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:15:28.922 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:15:28.923 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:15:28.923 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:15:28.925 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2023-11-24T06:15:28.928 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T06:15:28.928 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2023-11-24T06:15:28.928 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:15:28.928 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:15:28.928 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:15:28.928 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:15:28.929 INFO:teuthology.run_tasks:Running task install... 2023-11-24T06:15:28.937 DEBUG:teuthology.task.install:project ceph 2023-11-24T06:15:28.938 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}} 2023-11-24T06:15:28.938 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'} 2023-11-24T06:15:28.938 INFO:teuthology.task.install:Using flavor: default 2023-11-24T06:15:28.943 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-24T06:15:28.944 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-24T06:15:28.944 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'tag': None, 'wait_for_package': False} 2023-11-24T06:15:28.944 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:15:28.946 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'tag': None, 'wait_for_package': False} 2023-11-24T06:15:28.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:15:29.112 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/ 2023-11-24T06:15:29.113 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7484.ga13ea5ff 2023-11-24T06:15:29.167 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/ 2023-11-24T06:15:29.167 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7484.ga13ea5ff 2023-11-24T06:15:29.290 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-24T06:15:29.291 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:15:29.291 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-24T06:15:29.318 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-24T06:15:29.318 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a13ea5ff00cb4df7a9cc941166ddeb358881144c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-24T06:15:29.332 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-24T06:15:29.332 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:15:29.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-24T06:15:29.360 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-24T06:15:29.360 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a13ea5ff00cb4df7a9cc941166ddeb358881144c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-24T06:15:29.384 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-24T06:15:29.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-24T06:15:29.462 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-24T06:15:29.504 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-24T06:15:29.527 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2023-11-24T06:15:29.529 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2023-11-24T06:15:29.570 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2023-11-24T06:15:29.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2023-11-24T06:15:29.900 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-11-24T06:15:29.941 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2023-11-24T06:15:30.147 INFO:teuthology.orchestra.run.smithi089.stdout:44 files removed 2023-11-24T06:15:30.188 INFO:teuthology.orchestra.run.smithi123.stdout:44 files removed 2023-11-24T06:15:30.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-24T06:15:30.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-24T06:15:30.524 INFO:teuthology.orchestra.run.smithi089.stdout:Updating Subscription Management repositories. 2023-11-24T06:15:30.564 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2023-11-24T06:15:31.186 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 298 kB/s | 81 kB 00:00 2023-11-24T06:15:31.208 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 262 kB/s | 81 kB 00:00 2023-11-24T06:15:31.545 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 29 kB/s | 9.5 kB 00:00 2023-11-24T06:15:31.592 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 27 kB/s | 9.5 kB 00:00 2023-11-24T06:15:31.854 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 5.2 kB/s | 1.5 kB 00:00 2023-11-24T06:15:31.968 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 4.2 kB/s | 1.5 kB 00:00 2023-11-24T06:15:32.105 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-11-24T06:15:32.177 INFO:teuthology.orchestra.run.smithi089.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-11-24T06:15:32.956 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-11-24T06:15:36.489 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 379 kB/s | 24 kB 00:00 2023-11-24T06:15:37.423 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-11-24T06:15:44.909 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 16 MB 00:12 2023-11-24T06:15:45.692 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 54 MB/s | 47 MB 00:00 2023-11-24T06:15:48.417 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 517 kB/s | 24 kB 00:00 2023-11-24T06:15:49.261 INFO:teuthology.orchestra.run.smithi089.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 47 MB 00:00 2023-11-24T06:15:56.682 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-24T06:15:56.683 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-24T06:15:56.697 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-24T06:15:56.698 INFO:teuthology.orchestra.run.smithi123.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-24T06:15:56.699 INFO:teuthology.orchestra.run.smithi123.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-24T06:15:56.701 INFO:teuthology.orchestra.run.smithi123.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-24T06:15:56.703 INFO:teuthology.orchestra.run.smithi123.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-24T06:15:56.706 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-24T06:15:56.707 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-24T06:15:56.836 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-24T06:15:56.848 INFO:teuthology.orchestra.run.smithi123.stdout:=========================================================================================================================== 2023-11-24T06:15:56.848 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-11-24T06:15:56.848 INFO:teuthology.orchestra.run.smithi123.stdout:=========================================================================================================================== 2023-11-24T06:15:56.848 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-24T06:15:56.848 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-24T06:15:56.848 INFO:teuthology.orchestra.run.smithi123.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-24T06:15:56.848 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 6.4 k 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 5.4 M 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 920 k 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 171 k 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 1.6 M 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 133 k 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 1.8 M 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 7.4 M 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 50 k 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 14 M 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 49 M 2023-11-24T06:15:56.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 262 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 407 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 262 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 18 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 31 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 839 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 139 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-24T06:15:56.850 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 197 k 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 371 k 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 364 k 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 107 k 2023-11-24T06:15:56.851 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 94 k 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.6 M 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 181 k 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.8 M 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.7 M 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 23 M 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 28 k 2023-11-24T06:15:56.852 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 2.3 M 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 243 k 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 4.6 M 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 18 M 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 15 k 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 24 k 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-24T06:15:56.853 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 183 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 517 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 6.9 M 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-24T06:15:56.854 INFO:teuthology.orchestra.run.smithi123.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-24T06:15:56.855 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 45 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 128 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-24T06:15:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-24T06:15:56.857 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-24T06:15:56.858 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-24T06:15:56.859 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling module streams: 2023-11-24T06:15:56.860 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql 10 2023-11-24T06:15:56.861 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:15:56.861 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-24T06:15:56.861 INFO:teuthology.orchestra.run.smithi123.stdout:=========================================================================================================================== 2023-11-24T06:15:56.861 INFO:teuthology.orchestra.run.smithi123.stdout:Install 128 Packages 2023-11-24T06:15:56.861 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 3 Packages 2023-11-24T06:15:56.861 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:15:56.862 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 209 M 2023-11-24T06:15:56.862 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-24T06:15:57.146 INFO:teuthology.orchestra.run.smithi089.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 MB/s | 53 MB 00:00 2023-11-24T06:15:57.308 INFO:teuthology.orchestra.run.smithi123.stdout:(1/131): ceph-18.0.0-7484.ga13ea5ff.el8.x86_64. 35 kB/s | 6.4 kB 00:00 2023-11-24T06:15:57.567 INFO:teuthology.orchestra.run.smithi123.stdout:(2/131): ceph-base-18.0.0-7484.ga13ea5ff.el8.x8 12 MB/s | 5.4 MB 00:00 2023-11-24T06:15:57.617 INFO:teuthology.orchestra.run.smithi123.stdout:(3/131): ceph-immutable-object-cache-18.0.0-748 3.3 MB/s | 171 kB 00:00 2023-11-24T06:15:57.768 INFO:teuthology.orchestra.run.smithi123.stdout:(4/131): ceph-mds-18.0.0-7484.ga13ea5ff.el8.x86 15 MB/s | 2.3 MB 00:00 2023-11-24T06:15:57.876 INFO:teuthology.orchestra.run.smithi123.stdout:(5/131): ceph-mgr-18.0.0-7484.ga13ea5ff.el8.x86 15 MB/s | 1.6 MB 00:00 2023-11-24T06:15:58.161 INFO:teuthology.orchestra.run.smithi123.stdout:(6/131): ceph-mon-18.0.0-7484.ga13ea5ff.el8.x86 16 MB/s | 4.6 MB 00:00 2023-11-24T06:15:58.427 INFO:teuthology.orchestra.run.smithi123.stdout:(7/131): ceph-fuse-18.0.0-7484.ga13ea5ff.el8.x8 822 kB/s | 920 kB 00:01 2023-11-24T06:15:59.127 INFO:teuthology.orchestra.run.smithi123.stdout:(8/131): ceph-osd-18.0.0-7484.ga13ea5ff.el8.x86 19 MB/s | 18 MB 00:00 2023-11-24T06:15:59.177 INFO:teuthology.orchestra.run.smithi123.stdout:(9/131): ceph-selinux-18.0.0-7484.ga13ea5ff.el8 488 kB/s | 24 kB 00:00 2023-11-24T06:16:01.044 INFO:teuthology.orchestra.run.smithi123.stdout:(10/131): ceph-common-18.0.0-7484.ga13ea5ff.el8 5.8 MB/s | 23 MB 00:03 2023-11-24T06:16:01.111 INFO:teuthology.orchestra.run.smithi123.stdout:(11/131): cephfs-mirror-18.0.0-7484.ga13ea5ff.e 3.9 MB/s | 262 kB 00:00 2023-11-24T06:16:01.178 INFO:teuthology.orchestra.run.smithi123.stdout:(12/131): libcephfs-devel-18.0.0-7484.ga13ea5ff 467 kB/s | 31 kB 00:00 2023-11-24T06:16:01.337 INFO:teuthology.orchestra.run.smithi123.stdout:(13/131): libcephfs2-18.0.0-7484.ga13ea5ff.el8. 5.2 MB/s | 839 kB 00:00 2023-11-24T06:16:01.420 INFO:teuthology.orchestra.run.smithi123.stdout:(14/131): libcephsqlite-18.0.0-7484.ga13ea5ff.e 2.1 MB/s | 183 kB 00:00 2023-11-24T06:16:01.504 INFO:teuthology.orchestra.run.smithi123.stdout:(15/131): librados-devel-18.0.0-7484.ga13ea5ff. 1.6 MB/s | 139 kB 00:00 2023-11-24T06:16:01.604 INFO:teuthology.orchestra.run.smithi123.stdout:(16/131): libradosstriper1-18.0.0-7484.ga13ea5f 5.0 MB/s | 517 kB 00:00 2023-11-24T06:16:02.454 INFO:teuthology.orchestra.run.smithi123.stdout:(17/131): ceph-test-18.0.0-7484.ga13ea5ff.el8.x 15 MB/s | 49 MB 00:03 2023-11-24T06:16:02.575 INFO:teuthology.orchestra.run.smithi123.stdout:(18/131): librgw2-18.0.0-7484.ga13ea5ff.el8.x86 7.1 MB/s | 6.9 MB 00:00 2023-11-24T06:16:02.605 INFO:teuthology.orchestra.run.smithi123.stdout:(19/131): python3-ceph-argparse-18.0.0-7484.ga1 298 kB/s | 45 kB 00:00 2023-11-24T06:16:02.638 INFO:teuthology.orchestra.run.smithi123.stdout:(20/131): python3-ceph-common-18.0.0-7484.ga13e 2.0 MB/s | 128 kB 00:00 2023-11-24T06:16:02.664 INFO:teuthology.orchestra.run.smithi123.stdout:(21/131): python3-cephfs-18.0.0-7484.ga13ea5ff. 3.3 MB/s | 197 kB 00:00 2023-11-24T06:16:02.714 INFO:teuthology.orchestra.run.smithi123.stdout:(22/131): python3-rados-18.0.0-7484.ga13ea5ff.e 4.8 MB/s | 371 kB 00:00 2023-11-24T06:16:02.739 INFO:teuthology.orchestra.run.smithi123.stdout:(23/131): python3-rbd-18.0.0-7484.ga13ea5ff.el8 4.7 MB/s | 364 kB 00:00 2023-11-24T06:16:02.781 INFO:teuthology.orchestra.run.smithi123.stdout:(24/131): python3-rgw-18.0.0-7484.ga13ea5ff.el8 1.6 MB/s | 107 kB 00:00 2023-11-24T06:16:02.806 INFO:teuthology.orchestra.run.smithi123.stdout:(25/131): rbd-fuse-18.0.0-7484.ga13ea5ff.el8.x8 1.4 MB/s | 94 kB 00:00 2023-11-24T06:16:02.899 INFO:teuthology.orchestra.run.smithi123.stdout:(26/131): rbd-nbd-18.0.0-7484.ga13ea5ff.el8.x86 1.9 MB/s | 181 kB 00:00 2023-11-24T06:16:02.965 INFO:teuthology.orchestra.run.smithi123.stdout:(27/131): ceph-grafana-dashboards-18.0.0-7484.g 420 kB/s | 28 kB 00:00 2023-11-24T06:16:03.099 INFO:teuthology.orchestra.run.smithi123.stdout:(28/131): ceph-radosgw-18.0.0-7484.ga13ea5ff.el 3.0 MB/s | 14 MB 00:04 2023-11-24T06:16:03.125 INFO:teuthology.orchestra.run.smithi123.stdout:(29/131): ceph-mgr-cephadm-18.0.0-7484.ga13ea5f 836 kB/s | 133 kB 00:00 2023-11-24T06:16:03.183 INFO:teuthology.orchestra.run.smithi123.stdout:(30/131): rbd-mirror-18.0.0-7484.ga13ea5ff.el8. 8.9 MB/s | 3.6 MB 00:00 2023-11-24T06:16:03.308 INFO:teuthology.orchestra.run.smithi123.stdout:(31/131): ceph-mgr-modules-core-18.0.0-7484.ga1 1.9 MB/s | 243 kB 00:00 2023-11-24T06:16:03.417 INFO:teuthology.orchestra.run.smithi123.stdout:(32/131): ceph-mgr-rook-18.0.0-7484.ga13ea5ff.e 463 kB/s | 50 kB 00:00 2023-11-24T06:16:03.517 INFO:teuthology.orchestra.run.smithi123.stdout:(33/131): ceph-prometheus-alerts-18.0.0-7484.ga 150 kB/s | 15 kB 00:00 2023-11-24T06:16:03.643 INFO:teuthology.orchestra.run.smithi123.stdout:(34/131): ceph-volume-18.0.0-7484.ga13ea5ff.el8 2.0 MB/s | 262 kB 00:00 2023-11-24T06:16:03.776 INFO:teuthology.orchestra.run.smithi123.stdout:(35/131): ceph-mgr-diskprediction-local-18.0.0- 11 MB/s | 7.4 MB 00:00 2023-11-24T06:16:03.818 INFO:teuthology.orchestra.run.smithi123.stdout:(36/131): ceph-mgr-dashboard-18.0.0-7484.ga13ea 2.4 MB/s | 1.8 MB 00:00 2023-11-24T06:16:03.843 INFO:teuthology.orchestra.run.smithi123.stdout:(37/131): cephfs-top-18.0.0-7484.ga13ea5ff.el8. 271 kB/s | 18 kB 00:00 2023-11-24T06:16:03.869 INFO:teuthology.orchestra.run.smithi123.stdout:(38/131): cephadm-18.0.0-7484.ga13ea5ff.el8.noa 1.8 MB/s | 407 kB 00:00 2023-11-24T06:16:04.111 INFO:teuthology.orchestra.run.smithi123.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 220 kB/s | 59 kB 00:00 2023-11-24T06:16:04.136 INFO:teuthology.orchestra.run.smithi123.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 281 kB/s | 75 kB 00:00 2023-11-24T06:16:04.178 INFO:teuthology.orchestra.run.smithi123.stdout:(41/131): gperftools-libs-2.7-9.el8.x86_64.rpm 853 kB/s | 306 kB 00:00 2023-11-24T06:16:04.203 INFO:teuthology.orchestra.run.smithi123.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 652 kB/s | 44 kB 00:00 2023-11-24T06:16:04.245 INFO:teuthology.orchestra.run.smithi123.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2023-11-24T06:16:04.270 INFO:teuthology.orchestra.run.smithi123.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 494 kB/s | 33 kB 00:00 2023-11-24T06:16:04.304 INFO:teuthology.orchestra.run.smithi123.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 278 kB/s | 16 kB 00:00 2023-11-24T06:16:04.354 INFO:teuthology.orchestra.run.smithi123.stdout:(46/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.9 MB/s | 476 kB 00:00 2023-11-24T06:16:04.413 INFO:teuthology.orchestra.run.smithi123.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2023-11-24T06:16:04.438 INFO:teuthology.orchestra.run.smithi123.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.0 MB/s | 173 kB 00:00 2023-11-24T06:16:04.463 INFO:teuthology.orchestra.run.smithi123.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 750 kB/s | 81 kB 00:00 2023-11-24T06:16:04.480 INFO:teuthology.orchestra.run.smithi123.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 165 kB/s | 11 kB 00:00 2023-11-24T06:16:04.522 INFO:teuthology.orchestra.run.smithi123.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 301 kB/s | 18 kB 00:00 2023-11-24T06:16:04.539 INFO:teuthology.orchestra.run.smithi123.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 831 kB/s | 49 kB 00:00 2023-11-24T06:16:04.581 INFO:teuthology.orchestra.run.smithi123.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-11-24T06:16:04.598 INFO:teuthology.orchestra.run.smithi123.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2023-11-24T06:16:04.640 INFO:teuthology.orchestra.run.smithi123.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 6.6 MB/s | 1.2 MB 00:00 2023-11-24T06:16:04.657 INFO:teuthology.orchestra.run.smithi123.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2023-11-24T06:16:04.682 INFO:teuthology.orchestra.run.smithi123.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2023-11-24T06:16:04.707 INFO:teuthology.orchestra.run.smithi123.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 503 kB/s | 34 kB 00:00 2023-11-24T06:16:04.733 INFO:teuthology.orchestra.run.smithi123.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 2.5 MB/s | 196 kB 00:00 2023-11-24T06:16:04.749 INFO:teuthology.orchestra.run.smithi123.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 957 kB/s | 64 kB 00:00 2023-11-24T06:16:04.775 INFO:teuthology.orchestra.run.smithi123.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 278 kB/s | 19 kB 00:00 2023-11-24T06:16:04.800 INFO:teuthology.orchestra.run.smithi123.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 356 kB/s | 24 kB 00:00 2023-11-24T06:16:04.825 INFO:teuthology.orchestra.run.smithi123.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2023-11-24T06:16:04.851 INFO:teuthology.orchestra.run.smithi123.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 364 kB/s | 27 kB 00:00 2023-11-24T06:16:04.884 INFO:teuthology.orchestra.run.smithi123.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 3.0 MB/s | 255 kB 00:00 2023-11-24T06:16:04.910 INFO:teuthology.orchestra.run.smithi123.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.9 MB/s | 251 kB 00:00 2023-11-24T06:16:04.935 INFO:teuthology.orchestra.run.smithi123.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 729 kB/s | 61 kB 00:00 2023-11-24T06:16:04.960 INFO:teuthology.orchestra.run.smithi123.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2023-11-24T06:16:04.985 INFO:teuthology.orchestra.run.smithi123.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 305 kB/s | 23 kB 00:00 2023-11-24T06:16:05.011 INFO:teuthology.orchestra.run.smithi123.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-11-24T06:16:05.078 INFO:teuthology.orchestra.run.smithi123.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2023-11-24T06:16:05.103 INFO:teuthology.orchestra.run.smithi123.stdout:(72/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 400 kB/s | 47 kB 00:00 2023-11-24T06:16:05.136 INFO:teuthology.orchestra.run.smithi123.stdout:(73/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.6 MB/s | 200 kB 00:00 2023-11-24T06:16:05.162 INFO:teuthology.orchestra.run.smithi123.stdout:(74/131): python3-jwt-1.6.1-2.el8.noarch.rpm 509 kB/s | 43 kB 00:00 2023-11-24T06:16:05.187 INFO:teuthology.orchestra.run.smithi123.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 960 kB/s | 48 kB 00:00 2023-11-24T06:16:05.204 INFO:teuthology.orchestra.run.smithi123.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-11-24T06:16:05.229 INFO:teuthology.orchestra.run.smithi123.stdout:(77/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2023-11-24T06:16:05.280 INFO:teuthology.orchestra.run.smithi123.stdout:(78/131): python3-requests-oauthlib-1.0.0-1.el8 464 kB/s | 43 kB 00:00 2023-11-24T06:16:05.305 INFO:teuthology.orchestra.run.smithi123.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 1.1 MB/s | 109 kB 00:00 2023-11-24T06:16:05.330 INFO:teuthology.orchestra.run.smithi123.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2023-11-24T06:16:05.355 INFO:teuthology.orchestra.run.smithi123.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 373 kB/s | 19 kB 00:00 2023-11-24T06:16:05.381 INFO:teuthology.orchestra.run.smithi123.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-11-24T06:16:05.414 INFO:teuthology.orchestra.run.smithi123.stdout:(83/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.1 MB/s | 560 kB 00:00 2023-11-24T06:16:05.573 INFO:teuthology.orchestra.run.smithi123.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 93 kB/s | 18 kB 00:00 2023-11-24T06:16:05.607 INFO:teuthology.orchestra.run.smithi123.stdout:(85/131): python3-cryptography-3.2.1-5.el8.x86_ 2.2 MB/s | 559 kB 00:00 2023-11-24T06:16:05.632 INFO:teuthology.orchestra.run.smithi123.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-11-24T06:16:05.657 INFO:teuthology.orchestra.run.smithi123.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.4 MB/s | 605 kB 00:00 2023-11-24T06:16:05.682 INFO:teuthology.orchestra.run.smithi123.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.4 MB/s | 72 kB 00:00 2023-11-24T06:16:05.707 INFO:teuthology.orchestra.run.smithi123.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-11-24T06:16:05.733 INFO:teuthology.orchestra.run.smithi123.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-11-24T06:16:05.750 INFO:teuthology.orchestra.run.smithi123.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.3 MB/s | 98 kB 00:00 2023-11-24T06:16:05.775 INFO:teuthology.orchestra.run.smithi123.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.3 MB/s | 58 kB 00:00 2023-11-24T06:16:05.800 INFO:teuthology.orchestra.run.smithi123.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 767 kB/s | 39 kB 00:00 2023-11-24T06:16:05.835 INFO:teuthology.orchestra.run.smithi123.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 11 MB/s | 644 kB 00:00 2023-11-24T06:16:05.859 INFO:teuthology.orchestra.run.smithi123.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.8 MB/s | 171 kB 00:00 2023-11-24T06:16:05.885 INFO:teuthology.orchestra.run.smithi123.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.7 MB/s | 138 kB 00:00 2023-11-24T06:16:05.927 INFO:teuthology.orchestra.run.smithi123.stdout:(97/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 7.3 MB/s | 2.3 MB 00:00 2023-11-24T06:16:05.952 INFO:teuthology.orchestra.run.smithi123.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.6 MB/s | 110 kB 00:00 2023-11-24T06:16:05.969 INFO:teuthology.orchestra.run.smithi123.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-11-24T06:16:06.003 INFO:teuthology.orchestra.run.smithi123.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2023-11-24T06:16:06.036 INFO:teuthology.orchestra.run.smithi123.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-11-24T06:16:06.062 INFO:teuthology.orchestra.run.smithi123.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 663 kB/s | 39 kB 00:00 2023-11-24T06:16:06.095 INFO:teuthology.orchestra.run.smithi123.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-11-24T06:16:06.159 INFO:teuthology.orchestra.run.smithi123.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 5.4 MB/s | 538 kB 00:00 2023-11-24T06:16:06.187 INFO:teuthology.orchestra.run.smithi123.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2023-11-24T06:16:06.204 INFO:teuthology.orchestra.run.smithi123.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.1 MB/s | 187 kB 00:00 2023-11-24T06:16:06.255 INFO:teuthology.orchestra.run.smithi123.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-11-24T06:16:06.305 INFO:teuthology.orchestra.run.smithi123.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 533 kB/s | 27 kB 00:00 2023-11-24T06:16:06.355 INFO:teuthology.orchestra.run.smithi123.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-11-24T06:16:06.430 INFO:teuthology.orchestra.run.smithi123.stdout:(110/131): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-11-24T06:16:06.456 INFO:teuthology.orchestra.run.smithi123.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 439 kB/s | 44 kB 00:00 2023-11-24T06:16:06.514 INFO:teuthology.orchestra.run.smithi123.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 7.7 MB/s | 457 kB 00:00 2023-11-24T06:16:06.581 INFO:teuthology.orchestra.run.smithi123.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 25 MB/s | 3.7 MB 00:00 2023-11-24T06:16:06.606 INFO:teuthology.orchestra.run.smithi123.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.4 MB/s | 225 kB 00:00 2023-11-24T06:16:06.631 INFO:teuthology.orchestra.run.smithi123.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-11-24T06:16:06.657 INFO:teuthology.orchestra.run.smithi123.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-11-24T06:16:06.682 INFO:teuthology.orchestra.run.smithi123.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 397 kB/s | 20 kB 00:00 2023-11-24T06:16:06.708 INFO:teuthology.orchestra.run.smithi123.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-11-24T06:16:06.849 INFO:teuthology.orchestra.run.smithi123.stdout:(119/131): python3-scipy-1.0.0-21.module+el8.5. 14 MB/s | 14 MB 00:00 2023-11-24T06:16:06.883 INFO:teuthology.orchestra.run.smithi123.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2023-11-24T06:16:06.908 INFO:teuthology.orchestra.run.smithi123.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 3.3 MB/s | 198 kB 00:00 2023-11-24T06:16:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 3.9 MB/s | 202 kB 00:00 2023-11-24T06:16:07.068 INFO:teuthology.orchestra.run.smithi123.stdout:(123/131): openblas-threads-0.3.15-3.el8.x86_64 14 MB/s | 4.9 MB 00:00 2023-11-24T06:16:07.092 INFO:teuthology.orchestra.run.smithi123.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.6 MB/s | 354 kB 00:00 2023-11-24T06:16:07.160 INFO:teuthology.orchestra.run.smithi123.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-11-24T06:16:07.202 INFO:teuthology.orchestra.run.smithi123.stdout:(126/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-11-24T06:16:07.252 INFO:teuthology.orchestra.run.smithi123.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 16 MB/s | 1.5 MB 00:00 2023-11-24T06:16:07.335 INFO:teuthology.orchestra.run.smithi123.stdout:(128/131): postgresql-server-10.21-2.module+el8 19 MB/s | 5.1 MB 00:00 2023-11-24T06:16:07.402 INFO:teuthology.orchestra.run.smithi123.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-11-24T06:16:07.586 INFO:teuthology.orchestra.run.smithi123.stdout:(130/131): librbd1-18.0.0-7484.ga13ea5ff.el8.x8 11 MB/s | 3.7 MB 00:00 2023-11-24T06:16:07.978 INFO:teuthology.orchestra.run.smithi123.stdout:(131/131): librados2-18.0.0-7484.ga13ea5ff.el8. 4.9 MB/s | 3.8 MB 00:00 2023-11-24T06:16:07.981 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-24T06:16:07.981 INFO:teuthology.orchestra.run.smithi123.stdout:Total 19 MB/s | 209 MB 00:11 2023-11-24T06:16:08.427 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-24T06:16:08.428 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-24T06:16:08.442 INFO:teuthology.orchestra.run.smithi089.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-24T06:16:08.443 INFO:teuthology.orchestra.run.smithi089.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-24T06:16:08.444 INFO:teuthology.orchestra.run.smithi089.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-24T06:16:08.446 INFO:teuthology.orchestra.run.smithi089.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-24T06:16:08.448 INFO:teuthology.orchestra.run.smithi089.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-24T06:16:08.451 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-24T06:16:08.452 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-24T06:16:08.583 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================================================== 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================================================== 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 6.4 k 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 5.4 M 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 920 k 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 171 k 2023-11-24T06:16:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 1.6 M 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 133 k 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 1.8 M 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 7.4 M 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 50 k 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 14 M 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 49 M 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 262 k 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 407 k 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 262 k 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 18 k 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-24T06:16:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 31 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 839 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 139 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 197 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 371 k 2023-11-24T06:16:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 364 k 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 107 k 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 94 k 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.6 M 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 181 k 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.8 M 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.7 M 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-24T06:16:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 23 M 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 28 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 2.3 M 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 243 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 4.6 M 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 18 M 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 15 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 24 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-24T06:16:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 183 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 517 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 6.9 M 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-24T06:16:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-24T06:16:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-24T06:16:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-24T06:16:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-24T06:16:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 45 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 128 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-24T06:16:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-24T06:16:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-24T06:16:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling module streams: 2023-11-24T06:16:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql 10 2023-11-24T06:16:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:16:08.607 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-11-24T06:16:08.607 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================================================== 2023-11-24T06:16:08.607 INFO:teuthology.orchestra.run.smithi089.stdout:Install 128 Packages 2023-11-24T06:16:08.607 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 3 Packages 2023-11-24T06:16:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:16:08.608 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 209 M 2023-11-24T06:16:08.608 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-11-24T06:16:08.824 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-24T06:16:08.905 INFO:teuthology.orchestra.run.smithi089.stdout:(1/131): ceph-18.0.0-7484.ga13ea5ff.el8.x86_64. 36 kB/s | 6.4 kB 00:00 2023-11-24T06:16:08.943 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-24T06:16:08.943 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-24T06:16:09.221 INFO:teuthology.orchestra.run.smithi089.stdout:(2/131): ceph-fuse-18.0.0-7484.ga13ea5ff.el8.x8 2.8 MB/s | 920 kB 00:00 2023-11-24T06:16:09.296 INFO:teuthology.orchestra.run.smithi089.stdout:(3/131): ceph-immutable-object-cache-18.0.0-748 2.2 MB/s | 171 kB 00:00 2023-11-24T06:16:09.788 INFO:teuthology.orchestra.run.smithi089.stdout:(4/131): ceph-mds-18.0.0-7484.ga13ea5ff.el8.x86 4.7 MB/s | 2.3 MB 00:00 2023-11-24T06:16:09.863 INFO:teuthology.orchestra.run.smithi089.stdout:(5/131): ceph-base-18.0.0-7484.ga13ea5ff.el8.x8 4.7 MB/s | 5.4 MB 00:01 2023-11-24T06:16:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-24T06:16:10.076 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-24T06:16:10.164 INFO:teuthology.orchestra.run.smithi089.stdout:(6/131): ceph-mgr-18.0.0-7484.ga13ea5ff.el8.x86 4.3 MB/s | 1.6 MB 00:00 2023-11-24T06:16:10.481 INFO:teuthology.orchestra.run.smithi089.stdout:(7/131): ceph-common-18.0.0-7484.ga13ea5ff.el8. 13 MB/s | 23 MB 00:01 2023-11-24T06:16:10.556 INFO:teuthology.orchestra.run.smithi089.stdout:(8/131): ceph-mon-18.0.0-7484.ga13ea5ff.el8.x86 6.7 MB/s | 4.6 MB 00:00 2023-11-24T06:16:10.648 INFO:teuthology.orchestra.run.smithi089.stdout:(9/131): ceph-selinux-18.0.0-7484.ga13ea5ff.el8 265 kB/s | 24 kB 00:00 2023-11-24T06:16:11.781 INFO:teuthology.orchestra.run.smithi089.stdout:(10/131): ceph-radosgw-18.0.0-7484.ga13ea5ff.el 11 MB/s | 14 MB 00:01 2023-11-24T06:16:11.848 INFO:teuthology.orchestra.run.smithi089.stdout:(11/131): cephfs-mirror-18.0.0-7484.ga13ea5ff.e 3.8 MB/s | 262 kB 00:00 2023-11-24T06:16:11.882 INFO:teuthology.orchestra.run.smithi089.stdout:(12/131): libcephfs-devel-18.0.0-7484.ga13ea5ff 935 kB/s | 31 kB 00:00 2023-11-24T06:16:11.966 INFO:teuthology.orchestra.run.smithi089.stdout:(13/131): libcephfs2-18.0.0-7484.ga13ea5ff.el8. 9.8 MB/s | 839 kB 00:00 2023-11-24T06:16:12.007 INFO:teuthology.orchestra.run.smithi089.stdout:(14/131): libcephsqlite-18.0.0-7484.ga13ea5ff.e 4.3 MB/s | 183 kB 00:00 2023-11-24T06:16:12.058 INFO:teuthology.orchestra.run.smithi089.stdout:(15/131): librados-devel-18.0.0-7484.ga13ea5ff. 2.7 MB/s | 139 kB 00:00 2023-11-24T06:16:12.133 INFO:teuthology.orchestra.run.smithi089.stdout:(16/131): libradosstriper1-18.0.0-7484.ga13ea5f 6.7 MB/s | 517 kB 00:00 2023-11-24T06:16:12.775 INFO:teuthology.orchestra.run.smithi089.stdout:(17/131): librgw2-18.0.0-7484.ga13ea5ff.el8.x86 11 MB/s | 6.9 MB 00:00 2023-11-24T06:16:12.834 INFO:teuthology.orchestra.run.smithi089.stdout:(18/131): python3-ceph-argparse-18.0.0-7484.ga1 769 kB/s | 45 kB 00:00 2023-11-24T06:16:12.892 INFO:teuthology.orchestra.run.smithi089.stdout:(19/131): python3-ceph-common-18.0.0-7484.ga13e 2.2 MB/s | 128 kB 00:00 2023-11-24T06:16:12.959 INFO:teuthology.orchestra.run.smithi089.stdout:(20/131): python3-cephfs-18.0.0-7484.ga13ea5ff. 2.9 MB/s | 197 kB 00:00 2023-11-24T06:16:13.034 INFO:teuthology.orchestra.run.smithi089.stdout:(21/131): python3-rados-18.0.0-7484.ga13ea5ff.e 4.8 MB/s | 371 kB 00:00 2023-11-24T06:16:13.110 INFO:teuthology.orchestra.run.smithi089.stdout:(22/131): python3-rbd-18.0.0-7484.ga13ea5ff.el8 4.7 MB/s | 364 kB 00:00 2023-11-24T06:16:13.294 INFO:teuthology.orchestra.run.smithi089.stdout:(23/131): ceph-osd-18.0.0-7484.ga13ea5ff.el8.x8 5.8 MB/s | 18 MB 00:03 2023-11-24T06:16:13.319 INFO:teuthology.orchestra.run.smithi089.stdout:(24/131): python3-rgw-18.0.0-7484.ga13ea5ff.el8 512 kB/s | 107 kB 00:00 2023-11-24T06:16:13.353 INFO:teuthology.orchestra.run.smithi089.stdout:(25/131): rbd-fuse-18.0.0-7484.ga13ea5ff.el8.x8 1.6 MB/s | 94 kB 00:00 2023-11-24T06:16:13.461 INFO:teuthology.orchestra.run.smithi089.stdout:(26/131): rbd-nbd-18.0.0-7484.ga13ea5ff.el8.x86 1.6 MB/s | 181 kB 00:00 2023-11-24T06:16:13.561 INFO:teuthology.orchestra.run.smithi089.stdout:(27/131): ceph-grafana-dashboards-18.0.0-7484.g 279 kB/s | 28 kB 00:00 2023-11-24T06:16:13.645 INFO:teuthology.orchestra.run.smithi089.stdout:(28/131): rbd-mirror-18.0.0-7484.ga13ea5ff.el8. 11 MB/s | 3.6 MB 00:00 2023-11-24T06:16:13.670 INFO:teuthology.orchestra.run.smithi089.stdout:(29/131): ceph-mgr-cephadm-18.0.0-7484.ga13ea5f 1.2 MB/s | 133 kB 00:00 2023-11-24T06:16:13.879 INFO:teuthology.orchestra.run.smithi089.stdout:(30/131): ceph-mgr-dashboard-18.0.0-7484.ga13ea 7.5 MB/s | 1.8 MB 00:00 2023-11-24T06:16:13.946 INFO:teuthology.orchestra.run.smithi089.stdout:(31/131): ceph-mgr-modules-core-18.0.0-7484.ga1 3.6 MB/s | 243 kB 00:00 2023-11-24T06:16:14.004 INFO:teuthology.orchestra.run.smithi089.stdout:(32/131): ceph-mgr-rook-18.0.0-7484.ga13ea5ff.e 857 kB/s | 50 kB 00:00 2023-11-24T06:16:14.063 INFO:teuthology.orchestra.run.smithi089.stdout:(33/131): ceph-prometheus-alerts-18.0.0-7484.ga 257 kB/s | 15 kB 00:00 2023-11-24T06:16:14.147 INFO:teuthology.orchestra.run.smithi089.stdout:(34/131): ceph-volume-18.0.0-7484.ga13ea5ff.el8 3.1 MB/s | 262 kB 00:00 2023-11-24T06:16:14.222 INFO:teuthology.orchestra.run.smithi089.stdout:(35/131): cephadm-18.0.0-7484.ga13ea5ff.el8.noa 5.3 MB/s | 407 kB 00:00 2023-11-24T06:16:14.673 INFO:teuthology.orchestra.run.smithi089.stdout:(36/131): ceph-test-18.0.0-7484.ga13ea5ff.el8.x 12 MB/s | 49 MB 00:04 2023-11-24T06:16:14.714 INFO:teuthology.orchestra.run.smithi089.stdout:(37/131): cephfs-top-18.0.0-7484.ga13ea5ff.el8. 37 kB/s | 18 kB 00:00 2023-11-24T06:16:14.840 INFO:teuthology.orchestra.run.smithi089.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 6.3 MB/s | 7.4 MB 00:01 2023-11-24T06:16:14.923 INFO:teuthology.orchestra.run.smithi089.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2023-11-24T06:16:14.973 INFO:teuthology.orchestra.run.smithi089.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 567 kB/s | 75 kB 00:00 2023-11-24T06:16:15.007 INFO:teuthology.orchestra.run.smithi089.stdout:(41/131): gperftools-libs-2.7-9.el8.x86_64.rpm 916 kB/s | 306 kB 00:00 2023-11-24T06:16:15.033 INFO:teuthology.orchestra.run.smithi089.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 741 kB/s | 44 kB 00:00 2023-11-24T06:16:15.069 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-24T06:16:15.074 INFO:teuthology.orchestra.run.smithi089.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 795 kB/s | 33 kB 00:00 2023-11-24T06:16:15.108 INFO:teuthology.orchestra.run.smithi089.stdout:(44/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.5 MB/s | 476 kB 00:00 2023-11-24T06:16:15.133 INFO:teuthology.orchestra.run.smithi089.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2023-11-24T06:16:15.159 INFO:teuthology.orchestra.run.smithi089.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 195 kB/s | 16 kB 00:00 2023-11-24T06:16:15.200 INFO:teuthology.orchestra.run.smithi089.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.8 MB/s | 173 kB 00:00 2023-11-24T06:16:15.226 INFO:teuthology.orchestra.run.smithi089.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.2 MB/s | 81 kB 00:00 2023-11-24T06:16:15.251 INFO:teuthology.orchestra.run.smithi089.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-11-24T06:16:15.261 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-24T06:16:15.276 INFO:teuthology.orchestra.run.smithi089.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2023-11-24T06:16:15.302 INFO:teuthology.orchestra.run.smithi089.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.2 MB/s | 384 kB 00:00 2023-11-24T06:16:15.335 INFO:teuthology.orchestra.run.smithi089.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 833 kB/s | 49 kB 00:00 2023-11-24T06:16:15.361 INFO:teuthology.orchestra.run.smithi089.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-11-24T06:16:15.386 INFO:teuthology.orchestra.run.smithi089.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-11-24T06:16:15.428 INFO:teuthology.orchestra.run.smithi089.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 376 kB/s | 16 kB 00:00 2023-11-24T06:16:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-24T06:16:15.470 INFO:teuthology.orchestra.run.smithi089.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 811 kB/s | 34 kB 00:00 2023-11-24T06:16:15.495 INFO:teuthology.orchestra.run.smithi089.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2023-11-24T06:16:15.537 INFO:teuthology.orchestra.run.smithi089.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 1.5 MB/s | 64 kB 00:00 2023-11-24T06:16:15.562 INFO:teuthology.orchestra.run.smithi089.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2023-11-24T06:16:15.580 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-24T06:16:15.588 INFO:teuthology.orchestra.run.smithi089.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 372 kB/s | 19 kB 00:00 2023-11-24T06:16:15.613 INFO:teuthology.orchestra.run.smithi089.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2023-11-24T06:16:15.638 INFO:teuthology.orchestra.run.smithi089.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 584 kB/s | 29 kB 00:00 2023-11-24T06:16:15.664 INFO:teuthology.orchestra.run.smithi089.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2023-11-24T06:16:15.730 INFO:teuthology.orchestra.run.smithi089.stdout:(64/131): python3-kubernetes-11.0.0-6.el8.noarc 2.4 MB/s | 1.2 MB 00:00 2023-11-24T06:16:15.772 INFO:teuthology.orchestra.run.smithi089.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.9 MB/s | 255 kB 00:00 2023-11-24T06:16:15.798 INFO:teuthology.orchestra.run.smithi089.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 914 kB/s | 61 kB 00:00 2023-11-24T06:16:15.823 INFO:teuthology.orchestra.run.smithi089.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.5 MB/s | 251 kB 00:00 2023-11-24T06:16:15.848 INFO:teuthology.orchestra.run.smithi089.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 459 kB/s | 23 kB 00:00 2023-11-24T06:16:15.849 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-24T06:16:15.874 INFO:teuthology.orchestra.run.smithi089.stdout:(69/131): python3-webtest-2.0.33-1.el8.noarch.r 846 kB/s | 85 kB 00:00 2023-11-24T06:16:15.891 INFO:teuthology.orchestra.run.smithi089.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2023-11-24T06:16:15.949 INFO:teuthology.orchestra.run.smithi089.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.0 MB/s | 110 kB 00:00 2023-11-24T06:16:15.991 INFO:teuthology.orchestra.run.smithi089.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-11-24T06:16:16.025 INFO:teuthology.orchestra.run.smithi089.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.6 MB/s | 259 kB 00:00 2023-11-24T06:16:16.067 INFO:teuthology.orchestra.run.smithi089.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.5 MB/s | 323 kB 00:00 2023-11-24T06:16:16.096 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-24T06:16:16.109 INFO:teuthology.orchestra.run.smithi089.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 941 kB/s | 39 kB 00:00 2023-11-24T06:16:16.159 INFO:teuthology.orchestra.run.smithi089.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-11-24T06:16:16.218 INFO:teuthology.orchestra.run.smithi089.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 9.0 MB/s | 538 kB 00:00 2023-11-24T06:16:16.377 INFO:teuthology.orchestra.run.smithi089.stdout:(78/131): python3-scipy-1.0.0-21.module+el8.5.0 27 MB/s | 14 MB 00:00 2023-11-24T06:16:16.402 INFO:teuthology.orchestra.run.smithi089.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 854 kB/s | 157 kB 00:00 2023-11-24T06:16:16.427 INFO:teuthology.orchestra.run.smithi089.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-11-24T06:16:16.461 INFO:teuthology.orchestra.run.smithi089.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-11-24T06:16:16.492 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-24T06:16:16.494 INFO:teuthology.orchestra.run.smithi089.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 799 kB/s | 27 kB 00:00 2023-11-24T06:16:16.536 INFO:teuthology.orchestra.run.smithi089.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-11-24T06:16:16.612 INFO:teuthology.orchestra.run.smithi089.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-11-24T06:16:16.653 INFO:teuthology.orchestra.run.smithi089.stdout:(85/131): thrift-0.13.0-2.el8.x86_64.rpm 2.1 MB/s | 1.7 MB 00:00 2023-11-24T06:16:16.670 INFO:teuthology.orchestra.run.smithi089.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 328 kB/s | 44 kB 00:00 2023-11-24T06:16:16.712 INFO:teuthology.orchestra.run.smithi089.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 6.1 MB/s | 225 kB 00:00 2023-11-24T06:16:16.713 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 5/134 2023-11-24T06:16:16.738 INFO:teuthology.orchestra.run.smithi089.stdout:(88/131): python3-werkzeug-0.12.2-4.el8.noarch. 5.3 MB/s | 457 kB 00:00 2023-11-24T06:16:16.763 INFO:teuthology.orchestra.run.smithi089.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-11-24T06:16:16.788 INFO:teuthology.orchestra.run.smithi089.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-11-24T06:16:16.889 INFO:teuthology.orchestra.run.smithi089.stdout:(91/131): python3-numpy-1.14.3-10.el8.x86_64.rp 13 MB/s | 3.7 MB 00:00 2023-11-24T06:16:16.914 INFO:teuthology.orchestra.run.smithi089.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 133 kB/s | 20 kB 00:00 2023-11-24T06:16:16.939 INFO:teuthology.orchestra.run.smithi089.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 687 kB/s | 103 kB 00:00 2023-11-24T06:16:16.981 INFO:teuthology.orchestra.run.smithi089.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 7.5 MB/s | 709 kB 00:00 2023-11-24T06:16:17.006 INFO:teuthology.orchestra.run.smithi089.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.9 MB/s | 198 kB 00:00 2023-11-24T06:16:17.048 INFO:teuthology.orchestra.run.smithi089.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-11-24T06:16:17.090 INFO:teuthology.orchestra.run.smithi089.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-11-24T06:16:17.123 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 5/134 2023-11-24T06:16:17.166 INFO:teuthology.orchestra.run.smithi089.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 19 MB/s | 4.9 MB 00:00 2023-11-24T06:16:17.249 INFO:teuthology.orchestra.run.smithi089.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-11-24T06:16:17.283 INFO:teuthology.orchestra.run.smithi089.stdout:(100/131): postgresql-contrib-10.21-2.module+el 6.8 MB/s | 808 kB 00:00 2023-11-24T06:16:17.317 INFO:teuthology.orchestra.run.smithi089.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-11-24T06:16:17.321 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:16:17.358 INFO:teuthology.orchestra.run.smithi089.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 14 MB/s | 1.5 MB 00:00 2023-11-24T06:16:17.384 INFO:teuthology.orchestra.run.smithi089.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2023-11-24T06:16:17.409 INFO:teuthology.orchestra.run.smithi089.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 848 kB/s | 43 kB 00:00 2023-11-24T06:16:17.434 INFO:teuthology.orchestra.run.smithi089.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-11-24T06:16:17.457 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:16:17.459 INFO:teuthology.orchestra.run.smithi089.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 960 kB/s | 48 kB 00:00 2023-11-24T06:16:17.485 INFO:teuthology.orchestra.run.smithi089.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-11-24T06:16:17.510 INFO:teuthology.orchestra.run.smithi089.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 849 kB/s | 43 kB 00:00 2023-11-24T06:16:17.536 INFO:teuthology.orchestra.run.smithi089.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-11-24T06:16:17.561 INFO:teuthology.orchestra.run.smithi089.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-11-24T06:16:17.595 INFO:teuthology.orchestra.run.smithi089.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-11-24T06:16:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 374 kB/s | 19 kB 00:00 2023-11-24T06:16:17.621 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:16:17.637 INFO:teuthology.orchestra.run.smithi089.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.6 MB/s | 155 kB 00:00 2023-11-24T06:16:17.712 INFO:teuthology.orchestra.run.smithi089.stdout:(114/131): postgresql-server-10.21-2.module+el8 8.1 MB/s | 5.1 MB 00:00 2023-11-24T06:16:17.715 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:16:17.729 INFO:teuthology.orchestra.run.smithi089.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 194 kB/s | 18 kB 00:00 2023-11-24T06:16:17.793 INFO:teuthology.orchestra.run.smithi089.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 3.0 MB/s | 559 kB 00:00 2023-11-24T06:16:17.830 INFO:teuthology.orchestra.run.smithi089.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.0 MB/s | 605 kB 00:00 2023-11-24T06:16:17.855 INFO:teuthology.orchestra.run.smithi089.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.8 MB/s | 238 kB 00:00 2023-11-24T06:16:17.880 INFO:teuthology.orchestra.run.smithi089.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.4 MB/s | 72 kB 00:00 2023-11-24T06:16:17.905 INFO:teuthology.orchestra.run.smithi089.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-11-24T06:16:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:16:17.931 INFO:teuthology.orchestra.run.smithi089.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-11-24T06:16:17.956 INFO:teuthology.orchestra.run.smithi089.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-11-24T06:16:17.981 INFO:teuthology.orchestra.run.smithi089.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-11-24T06:16:18.007 INFO:teuthology.orchestra.run.smithi089.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 766 kB/s | 39 kB 00:00 2023-11-24T06:16:18.040 INFO:teuthology.orchestra.run.smithi089.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 11 MB/s | 644 kB 00:00 2023-11-24T06:16:18.066 INFO:teuthology.orchestra.run.smithi089.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.8 MB/s | 171 kB 00:00 2023-11-24T06:16:18.089 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-11-24T06:16:18.091 INFO:teuthology.orchestra.run.smithi089.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-11-24T06:16:18.141 INFO:teuthology.orchestra.run.smithi089.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.7 MB/s | 2.3 MB 00:00 2023-11-24T06:16:18.183 INFO:teuthology.orchestra.run.smithi089.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 19 MB/s | 798 kB 00:00 2023-11-24T06:16:18.325 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-24T06:16:18.516 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-24T06:16:18.583 INFO:teuthology.orchestra.run.smithi089.stdout:(130/131): librbd1-18.0.0-7484.ga13ea5ff.el8.x8 7.5 MB/s | 3.7 MB 00:00 2023-11-24T06:16:18.684 INFO:teuthology.orchestra.run.smithi089.stdout:(131/131): librados2-18.0.0-7484.ga13ea5ff.el8. 6.1 MB/s | 3.8 MB 00:00 2023-11-24T06:16:18.687 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-11-24T06:16:18.687 INFO:teuthology.orchestra.run.smithi089.stdout:Total 21 MB/s | 209 MB 00:10 2023-11-24T06:16:18.929 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-24T06:16:19.170 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-24T06:16:19.329 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 14/134 2023-11-24T06:16:19.459 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 14/134 2023-11-24T06:16:19.513 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-11-24T06:16:19.613 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 15/134 2023-11-24T06:16:19.632 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-11-24T06:16:19.633 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-11-24T06:16:19.777 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 15/134 2023-11-24T06:16:19.949 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-11-24T06:16:20.118 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-11-24T06:16:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-11-24T06:16:20.387 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-24T06:16:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-24T06:16:20.671 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-24T06:16:20.762 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-11-24T06:16:20.766 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-11-24T06:16:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-24T06:16:21.796 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-24T06:16:22.398 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-24T06:16:22.570 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-11-24T06:16:23.277 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-11-24T06:16:23.542 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-11-24T06:16:23.734 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-11-24T06:16:23.948 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-24T06:16:24.110 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8. 26/134 2023-11-24T06:16:24.248 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 27/134 2023-11-24T06:16:24.482 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_6 28/134 2023-11-24T06:16:24.670 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-24T06:16:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-11-24T06:16:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-24T06:16:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-24T06:16:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-24T06:16:26.418 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-24T06:16:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-24T06:16:26.628 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-24T06:16:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-24T06:16:27.245 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-24T06:16:27.397 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-11-24T06:16:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:16:27.692 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-11-24T06:16:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:16:27.971 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-11-24T06:16:28.078 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:16:28.167 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-11-24T06:16:28.215 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:16:28.348 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-24T06:16:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:16:28.481 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:16:28.547 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-24T06:16:28.711 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 9/134 2023-11-24T06:16:28.742 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 36/134 2023-11-24T06:16:28.877 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_6 37/134 2023-11-24T06:16:29.005 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-24T06:16:29.018 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-11-24T06:16:29.138 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-24T06:16:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-11-24T06:16:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-11-24T06:16:29.388 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-11-24T06:16:29.544 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-24T06:16:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-11-24T06:16:29.768 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-24T06:16:29.798 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-11-24T06:16:29.918 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 14/134 2023-11-24T06:16:29.953 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-24T06:16:30.017 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-24T06:16:30.025 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 14/134 2023-11-24T06:16:30.202 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 15/134 2023-11-24T06:16:30.206 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-11-24T06:16:30.284 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 15/134 2023-11-24T06:16:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-11-24T06:16:30.409 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-24T06:16:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-11-24T06:16:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-24T06:16:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-24T06:16:30.695 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-11-24T06:16:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-11-24T06:16:30.981 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-11-24T06:16:31.149 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-11-24T06:16:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-24T06:16:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-11-24T06:16:31.468 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-24T06:16:31.693 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-11-24T06:16:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-11-24T06:16:32.068 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-24T06:16:32.205 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-11-24T06:16:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-24T06:16:32.473 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-11-24T06:16:32.674 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-11-24T06:16:32.961 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-24T06:16:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-11-24T06:16:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-11-24T06:16:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-11-24T06:16:33.554 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-24T06:16:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-11-24T06:16:33.717 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-11-24T06:16:33.872 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-11-24T06:16:34.017 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-24T06:16:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-11-24T06:16:34.239 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-24T06:16:34.391 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-24T06:16:34.448 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-11-24T06:16:34.602 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-24T06:16:34.665 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-11-24T06:16:34.830 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-24T06:16:34.870 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-11-24T06:16:34.983 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8. 26/134 2023-11-24T06:16:35.059 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-24T06:16:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 27/134 2023-11-24T06:16:35.221 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-24T06:16:35.485 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-24T06:16:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-24T06:16:35.880 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-24T06:16:35.991 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_6 28/134 2023-11-24T06:16:36.011 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-24T06:16:36.200 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-24T06:16:36.323 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-24T06:16:36.395 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-24T06:16:36.569 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-24T06:16:36.593 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-24T06:16:36.753 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-24T06:16:36.882 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-24T06:16:36.907 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-24T06:16:37.076 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-24T06:16:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-24T06:16:37.254 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-24T06:16:37.440 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-24T06:16:37.667 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-24T06:16:37.900 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-24T06:16:38.048 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-24T06:16:38.257 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-24T06:16:38.867 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-24T06:16:39.226 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-24T06:16:39.333 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-24T06:16:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-24T06:16:39.605 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-24T06:16:39.627 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-24T06:16:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 36/134 2023-11-24T06:16:39.825 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-24T06:16:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_6 37/134 2023-11-24T06:16:40.127 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-24T06:16:40.138 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-24T06:16:40.318 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-24T06:16:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-24T06:16:40.494 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-24T06:16:40.569 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-24T06:16:40.629 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-24T06:16:40.816 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-24T06:16:41.047 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-24T06:16:43.823 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-24T06:16:43.824 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 92/134 2023-11-24T06:16:43.824 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 92/134 2023-11-24T06:16:43.824 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 93/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-24T06:16:43.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-24T06:16:43.915 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-24T06:16:44.178 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-24T06:16:44.403 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-24T06:16:44.622 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-24T06:16:44.781 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-24T06:16:44.917 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-24T06:16:45.091 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-24T06:16:45.160 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-24T06:16:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-24T06:16:45.458 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:16:45.636 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-24T06:16:45.667 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:16:45.791 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-24T06:16:45.871 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-24T06:16:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-24T06:16:46.241 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-24T06:16:46.248 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:16:46.382 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-24T06:16:46.523 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 95/134 2023-11-24T06:16:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-24T06:16:46.621 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 95/134 2023-11-24T06:16:46.758 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 96/134 2023-11-24T06:16:46.783 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-24T06:16:46.997 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-24T06:16:47.153 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-24T06:16:47.292 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-24T06:16:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-24T06:16:47.693 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-24T06:16:47.867 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-24T06:16:48.027 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-24T06:16:48.154 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-24T06:16:48.361 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-24T06:16:48.585 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-24T06:16:48.763 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-24T06:16:49.022 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-24T06:16:49.265 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-24T06:16:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-24T06:16:49.666 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-24T06:16:50.732 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-24T06:16:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-24T06:16:51.164 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-24T06:16:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-24T06:16:51.750 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-24T06:16:51.939 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-24T06:16:52.753 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-24T06:16:52.949 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 92/134 2023-11-24T06:16:53.024 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 92/134 2023-11-24T06:16:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 93/134 2023-11-24T06:16:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:16:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:16:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:16:58.059 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 95/134 2023-11-24T06:16:58.142 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 95/134 2023-11-24T06:16:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 96/134 2023-11-24T06:17:05.198 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 96/134 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /run 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2023-11-24T06:17:05.199 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:05.522 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 97/134 2023-11-24T06:17:07.118 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 97/134 2023-11-24T06:17:07.286 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 98/134 2023-11-24T06:17:07.513 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 98/134 2023-11-24T06:17:07.655 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 99/134 2023-11-24T06:17:08.113 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 99/134 2023-11-24T06:17:08.305 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 100/134 2023-11-24T06:17:08.322 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 100/134 2023-11-24T06:17:10.007 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 101/134 2023-11-24T06:17:10.196 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 101/134 2023-11-24T06:17:10.312 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 102/134 2023-11-24T06:17:10.520 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 103/134 2023-11-24T06:17:10.704 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 104/134 2023-11-24T06:17:10.926 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 104/134 2023-11-24T06:17:11.168 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 105/134 2023-11-24T06:17:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 105/134 2023-11-24T06:17:12.296 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 106/134 2023-11-24T06:17:12.386 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 106/134 2023-11-24T06:17:12.726 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 107/134 2023-11-24T06:17:12.893 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 108/134 2023-11-24T06:17:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 108/134 2023-11-24T06:17:13.095 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 109/134 2023-11-24T06:17:13.106 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 109/134 2023-11-24T06:17:14.439 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 110/134 2023-11-24T06:17:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 111/134 2023-11-24T06:17:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 111/134 2023-11-24T06:17:14.812 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 112/134 2023-11-24T06:17:14.855 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 112/134 2023-11-24T06:17:15.012 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 113/134 2023-11-24T06:17:15.370 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 113/134 2023-11-24T06:17:15.504 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 114/134 2023-11-24T06:17:16.710 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 96/134 2023-11-24T06:17:16.710 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2023-11-24T06:17:16.710 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2023-11-24T06:17:16.710 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /dev 2023-11-24T06:17:16.710 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /run 2023-11-24T06:17:16.711 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2023-11-24T06:17:16.711 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2023-11-24T06:17:16.711 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2023-11-24T06:17:16.711 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2023-11-24T06:17:16.711 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /dev 2023-11-24T06:17:16.711 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 97/134 2023-11-24T06:17:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 97/134 2023-11-24T06:17:18.822 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 98/134 2023-11-24T06:17:19.056 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 98/134 2023-11-24T06:17:19.195 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 99/134 2023-11-24T06:17:19.656 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 99/134 2023-11-24T06:17:19.844 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 100/134 2023-11-24T06:17:19.861 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 100/134 2023-11-24T06:17:20.154 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 114/134 2023-11-24T06:17:21.053 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 115/134 2023-11-24T06:17:21.382 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-11-24T06:17:21.543 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 117/134 2023-11-24T06:17:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 101/134 2023-11-24T06:17:21.805 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-11-24T06:17:21.816 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 101/134 2023-11-24T06:17:21.943 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 102/134 2023-11-24T06:17:21.952 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-11-24T06:17:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 103/134 2023-11-24T06:17:22.142 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-11-24T06:17:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 104/134 2023-11-24T06:17:22.357 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-11-24T06:17:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 104/134 2023-11-24T06:17:22.515 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_ 121/134 2023-11-24T06:17:22.645 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-24T06:17:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 105/134 2023-11-24T06:17:23.769 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 105/134 2023-11-24T06:17:23.918 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 106/134 2023-11-24T06:17:24.009 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 106/134 2023-11-24T06:17:24.381 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 107/134 2023-11-24T06:17:24.564 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 108/134 2023-11-24T06:17:24.601 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 108/134 2023-11-24T06:17:24.751 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 109/134 2023-11-24T06:17:24.763 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 109/134 2023-11-24T06:17:24.830 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:17:25.053 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:17:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:17:25.219 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 124/134 2023-11-24T06:17:25.461 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 125/134 2023-11-24T06:17:25.750 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 126/134 2023-11-24T06:17:26.079 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 110/134 2023-11-24T06:17:26.145 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-24T06:17:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 111/134 2023-11-24T06:17:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 111/134 2023-11-24T06:17:26.318 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-11-24T06:17:26.401 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-11-24T06:17:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 112/134 2023-11-24T06:17:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 112/134 2023-11-24T06:17:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-11-24T06:17:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-11-24T06:17:26.652 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 113/134 2023-11-24T06:17:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-24T06:17:26.802 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-24T06:17:26.937 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-11-24T06:17:26.937 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:17:27.010 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 113/134 2023-11-24T06:17:27.096 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:17:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-24T06:17:27.177 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 114/134 2023-11-24T06:17:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-24T06:17:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:17:28.006 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:17:28.006 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 1/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 3/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 4/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 5/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 9/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 10/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 11/134 2023-11-24T06:17:28.007 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 12/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 13/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_ 14/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 15/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 16/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_6 17/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 18/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 19/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 20/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8. 21/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_6 22/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 23/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 24/134 2023-11-24T06:17:28.008 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 25/134 2023-11-24T06:17:28.009 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 26/134 2023-11-24T06:17:28.009 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 27/134 2023-11-24T06:17:28.009 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 28/134 2023-11-24T06:17:28.009 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 29/134 2023-11-24T06:17:28.009 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 30/134 2023-11-24T06:17:28.018 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 31/134 2023-11-24T06:17:28.018 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 32/134 2023-11-24T06:17:28.018 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 33/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 34/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 35/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 36/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 37/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 38/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-24T06:17:28.019 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-24T06:17:28.020 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-24T06:17:28.021 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-11-24T06:17:28.022 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-11-24T06:17:28.023 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-11-24T06:17:28.023 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-11-24T06:17:28.023 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-11-24T06:17:28.023 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-11-24T06:17:28.023 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-11-24T06:17:28.023 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-11-24T06:17:28.023 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-11-24T06:17:28.027 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-11-24T06:17:28.027 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-11-24T06:17:28.027 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-11-24T06:17:28.027 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-11-24T06:17:28.027 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-11-24T06:17:28.028 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-11-24T06:17:28.029 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 129/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-24T06:17:28.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 131/134 2023-11-24T06:17:28.031 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:17:28.031 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-24T06:17:30.603 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:17:30.603 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:45 ago on Fri 24 Nov 2023 06:15:45 AM UTC. 2023-11-24T06:17:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 114/134 2023-11-24T06:17:32.155 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 115/134 2023-11-24T06:17:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-24T06:17:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-24T06:17:33.350 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-24T06:17:33.670 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-24T06:17:33.770 INFO:teuthology.orchestra.run.smithi123.stdout:Installed products updated. 2023-11-24T06:17:33.846 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 119/134 2023-11-24T06:17:33.994 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: automake-1.16.1-7.el8.noarch 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: bison-3.0.4-10.el8.x86_64 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.995 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: flex-2.6.1-9.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: jq-1.6-3.el8.x86_64 2023-11-24T06:17:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: libpq-13.5-1.el8.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-24T06:17:33.998 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-24T06:17:33.999 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-24T06:17:34.000 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-24T06:17:34.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-24T06:17:34.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-2.6-1.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-24T06:17:34.004 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-24T06:17:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-24T06:17:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-24T06:17:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:34.006 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-24T06:17:34.037 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-24T06:17:34.179 DEBUG:teuthology.parallel:result is None 2023-11-24T06:17:34.195 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_ 121/134 2023-11-24T06:17:34.343 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-24T06:17:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:17:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:17:36.834 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:17:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 124/134 2023-11-24T06:17:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 125/134 2023-11-24T06:17:37.548 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 126/134 2023-11-24T06:17:37.860 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-24T06:17:37.999 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-24T06:17:38.008 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-24T06:17:38.188 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-24T06:17:38.357 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-24T06:17:38.442 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-24T06:17:38.624 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-24T06:17:38.637 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-24T06:17:38.637 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:17:38.819 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:17:38.819 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-24T06:17:39.062 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-24T06:17:39.273 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:17:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:17:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 1/134 2023-11-24T06:17:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 3/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 4/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 5/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 9/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 10/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 11/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 12/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 13/134 2023-11-24T06:17:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_ 14/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 15/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 16/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_6 17/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 18/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 19/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 20/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8. 21/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_6 22/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 23/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 24/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 25/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 26/134 2023-11-24T06:17:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 27/134 2023-11-24T06:17:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 28/134 2023-11-24T06:17:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 29/134 2023-11-24T06:17:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 30/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 31/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 32/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 33/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 34/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 35/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 36/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 37/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 38/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-24T06:17:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-24T06:17:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-24T06:17:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-24T06:17:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-24T06:17:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-24T06:17:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-24T06:17:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-24T06:17:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 129/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 131/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:17:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-24T06:17:42.520 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:17:42.520 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:46 ago on Fri 24 Nov 2023 06:15:56 AM UTC. 2023-11-24T06:17:45.441 INFO:teuthology.orchestra.run.smithi089.stdout:Installed products updated. 2023-11-24T06:17:45.655 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:45.655 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2023-11-24T06:17:45.655 INFO:teuthology.orchestra.run.smithi089.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: automake-1.16.1-7.el8.noarch 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: bison-3.0.4-10.el8.x86_64 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.656 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.657 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: flex-2.6.1-9.el8.x86_64 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-24T06:17:45.658 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-3.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-24T06:17:45.659 INFO:teuthology.orchestra.run.smithi089.stdout: libpq-13.5-1.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-24T06:17:45.660 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:17:45.661 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.662 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-24T06:17:45.663 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-2.6-1.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-24T06:17:45.664 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.665 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-24T06:17:45.666 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-24T06:17:45.667 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.667 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.667 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:17:45.667 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-24T06:17:45.667 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-24T06:17:45.667 INFO:teuthology.orchestra.run.smithi089.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-24T06:17:45.667 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-24T06:17:45.667 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-24T06:17:45.668 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-24T06:17:45.668 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:45.668 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-11-24T06:17:45.853 DEBUG:teuthology.parallel:result is None 2023-11-24T06:17:45.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:17:46.081 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-24T06:17:46.104 INFO:teuthology.orchestra.run.smithi089.stdout:18.0.0-7484.ga13ea5ff.el8 2023-11-24T06:17:46.105 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7484.ga13ea5ff.el8 2023-11-24T06:17:46.105 INFO:teuthology.task.install:The correct ceph version 18.0.0-7484.ga13ea5ff is installed. 2023-11-24T06:17:46.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:17:46.245 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-24T06:17:46.274 INFO:teuthology.orchestra.run.smithi123.stdout:18.0.0-7484.ga13ea5ff.el8 2023-11-24T06:17:46.274 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7484.ga13ea5ff.el8 2023-11-24T06:17:46.274 INFO:teuthology.task.install:The correct ceph version 18.0.0-7484.ga13ea5ff is installed. 2023-11-24T06:17:46.274 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-24T06:17:46.274 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:46.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-24T06:17:46.315 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:17:46.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-24T06:17:46.365 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-24T06:17:46.365 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:46.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2023-11-24T06:17:46.393 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-24T06:17:46.457 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:17:46.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2023-11-24T06:17:46.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-24T06:17:46.559 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-24T06:17:46.559 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:46.559 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-24T06:17:46.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-24T06:17:46.651 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:17:46.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-24T06:17:46.680 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-24T06:17:46.748 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-24T06:17:46.748 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:46.748 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2023-11-24T06:17:46.774 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-24T06:17:46.840 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:17:46.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2023-11-24T06:17:46.869 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-24T06:17:46.938 INFO:teuthology.run_tasks:Running task ceph... 2023-11-24T06:17:47.009 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-24T06:17:47.010 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /var/log/ceph 2023-11-24T06:17:47.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /var/log/ceph 2023-11-24T06:17:47.040 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-24T06:17:47.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-24T06:17:47.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-24T06:17:47.108 INFO:tasks.ceph:Creating extra log directories... 2023-11-24T06:17:47.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-24T06:17:47.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-24T06:17:47.181 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-24T06:17:47.181 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-24T06:17:47.181 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-24T06:17:47.182 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:47.182 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-24T06:17:47.226 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-24T06:17:47.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-24T06:17:47.381 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:17:47.382 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-24T06:17:47.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-24T06:17:47.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-24T06:17:47.716 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-24T06:17:47.717 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-24T06:17:47.717 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556', 'branch': 'rishabh-mgr-vol-misc', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}', 'email': None, 'first_in_suite': False, 'job_id': '7465556', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'rishabh-mgr-vol-misc'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-mgr-vol-misc', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3305, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-mgr-vol-misc', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi123.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestVolumes', 'tasks.cephfs.test_volumes.TestSubvolumeGroups', 'tasks.cephfs.test_volumes.TestSubvolumes', 'tasks.cephfs.test_subvolume.TestSubvolume']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '2442239d2653456406c25ae0c71b689c8f2657b6', 'timestamp': '2023-11-24_05:41:33', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-24T06:17:47.717 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-24T06:17:47.720 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-24T06:17:47.769 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T06:17:47.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T06:17:47.837 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:47.837 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2023-11-24T06:17:47.859 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-24T06:17:47.859 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2023-11-24T06:17:47.916 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-24T06:17:47.916 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:17:47.916 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 6h/6d Inode: 385593 Links: 1 2023-11-24T06:17:47.916 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:17:47.916 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:17:47.916 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-11-24 06:17:39.489885080 +0000 2023-11-24T06:17:47.916 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-11-24 06:14:56.166673170 +0000 2023-11-24T06:17:47.917 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-11-24 06:14:56.166673170 +0000 2023-11-24T06:17:47.917 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: - 2023-11-24T06:17:47.917 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-24T06:17:47.983 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-11-24T06:17:47.983 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-11-24T06:17:47.983 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000313385 s, 1.6 MB/s 2023-11-24T06:17:47.984 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-24T06:17:48.043 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2023-11-24T06:17:48.100 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-24T06:17:48.100 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:17:48.100 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 6h/6d Inode: 385672 Links: 1 2023-11-24T06:17:48.100 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:17:48.101 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:17:48.101 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-11-24 06:17:39.489885080 +0000 2023-11-24T06:17:48.101 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-11-24 06:14:56.647661737 +0000 2023-11-24T06:17:48.101 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-11-24 06:14:56.647661737 +0000 2023-11-24T06:17:48.101 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: - 2023-11-24T06:17:48.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-24T06:17:48.166 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-11-24T06:17:48.166 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-11-24T06:17:48.166 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000244879 s, 2.1 MB/s 2023-11-24T06:17:48.167 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-24T06:17:48.226 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2023-11-24T06:17:48.283 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-24T06:17:48.283 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:17:48.283 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 6h/6d Inode: 381865 Links: 1 2023-11-24T06:17:48.283 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:17:48.283 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:17:48.284 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-11-24 06:17:39.489885080 +0000 2023-11-24T06:17:48.284 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-11-24 06:14:57.129650280 +0000 2023-11-24T06:17:48.284 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-11-24 06:14:57.129650280 +0000 2023-11-24T06:17:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: - 2023-11-24T06:17:48.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-24T06:17:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-11-24T06:17:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-11-24T06:17:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000223086 s, 2.3 MB/s 2023-11-24T06:17:48.349 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-24T06:17:48.406 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2023-11-24T06:17:48.463 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-24T06:17:48.463 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:17:48.463 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 6h/6d Inode: 381947 Links: 1 2023-11-24T06:17:48.464 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:17:48.464 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:17:48.464 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-11-24 06:17:39.489885080 +0000 2023-11-24T06:17:48.464 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-11-24 06:14:57.630638372 +0000 2023-11-24T06:17:48.464 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-11-24 06:14:57.630638372 +0000 2023-11-24T06:17:48.464 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: - 2023-11-24T06:17:48.464 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-24T06:17:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-11-24T06:17:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-11-24T06:17:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000193633 s, 2.6 MB/s 2023-11-24T06:17:48.530 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-24T06:17:48.587 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:17:48.587 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:17:48.587 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2023-11-24T06:17:48.609 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-24T06:17:48.609 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2023-11-24T06:17:48.668 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-24T06:17:48.669 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:17:48.669 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 384818 Links: 1 2023-11-24T06:17:48.669 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:17:48.669 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:17:48.669 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-11-24 06:17:27.702191018 +0000 2023-11-24T06:17:48.669 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-11-24 06:15:00.344729101 +0000 2023-11-24T06:17:48.669 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-11-24 06:15:00.344729101 +0000 2023-11-24T06:17:48.669 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-11-24T06:17:48.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-24T06:17:48.736 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-11-24T06:17:48.737 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-11-24T06:17:48.737 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000259335 s, 2.0 MB/s 2023-11-24T06:17:48.738 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-24T06:17:48.798 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2023-11-24T06:17:48.856 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-24T06:17:48.856 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:17:48.856 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 392623 Links: 1 2023-11-24T06:17:48.856 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:17:48.856 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:17:48.856 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-11-24 06:17:27.702191018 +0000 2023-11-24T06:17:48.856 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-11-24 06:15:00.864716616 +0000 2023-11-24T06:17:48.856 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-11-24 06:15:00.864716616 +0000 2023-11-24T06:17:48.857 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-11-24T06:17:48.857 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-24T06:17:48.922 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-11-24T06:17:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-11-24T06:17:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000228979 s, 2.2 MB/s 2023-11-24T06:17:48.924 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-24T06:17:48.982 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 391703 Links: 1 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-11-24 06:17:27.702191018 +0000 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-11-24 06:15:01.357704779 +0000 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-11-24 06:15:01.357704779 +0000 2023-11-24T06:17:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-11-24T06:17:49.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-24T06:17:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-11-24T06:17:49.109 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-11-24T06:17:49.109 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000269594 s, 1.9 MB/s 2023-11-24T06:17:49.110 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-24T06:17:49.167 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2023-11-24T06:17:49.224 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-24T06:17:49.225 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:17:49.225 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 395265 Links: 1 2023-11-24T06:17:49.225 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:17:49.225 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:17:49.225 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-11-24 06:17:27.702191018 +0000 2023-11-24T06:17:49.225 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-11-24 06:15:01.857692774 +0000 2023-11-24T06:17:49.225 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-11-24 06:15:01.857692774 +0000 2023-11-24T06:17:49.225 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-11-24T06:17:49.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-24T06:17:49.290 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-11-24T06:17:49.290 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-11-24T06:17:49.290 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000217446 s, 2.4 MB/s 2023-11-24T06:17:49.291 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-24T06:17:49.349 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:17:49.349 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-24T06:17:49.349 INFO:tasks.ceph:Generating config... 2023-11-24T06:17:49.350 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-24T06:17:49.350 INFO:tasks.ceph:[client] debug client = 20 2023-11-24T06:17:49.351 INFO:tasks.ceph:[client] debug ms = 1 2023-11-24T06:17:49.351 INFO:tasks.ceph:[client] pid file = 2023-11-24T06:17:49.351 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-24T06:17:49.351 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-24T06:17:49.351 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-11-24T06:17:49.351 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-24T06:17:49.351 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-24T06:17:49.351 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-24T06:17:49.351 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-24T06:17:49.351 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-24T06:17:49.351 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-24T06:17:49.351 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mgr] debug client = 20 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-24T06:17:49.352 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-24T06:17:49.352 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-24T06:17:49.352 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-24T06:17:49.353 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-24T06:17:49.354 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-24T06:17:49.354 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-24T06:17:49.354 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-24T06:17:49.354 INFO:tasks.ceph:Setting up mon.a... 2023-11-24T06:17:49.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-24T06:17:49.410 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.keyring 2023-11-24T06:17:49.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-24T06:17:49.462 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T06:17:49.490 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.89'), ('mon.b', '172.21.15.123'), ('mon.c', '[v2:172.21.15.123:3301,v1:172.21.15.123:6790]')] 2023-11-24T06:17:49.491 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.89,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-24T06:17:49.492 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:49.492 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T06:17:49.546 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.89 --add b 172.21.15.123 --addv c '[v2:172.21.15.123:3301,v1:172.21.15.123:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:17:49.635 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:17:49.635 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: generated fsid be2488f4-68a7-42cc-93d1-ae9c6921fb48 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = pacific 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 0 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:fsid be2488f4-68a7-42cc-93d1-ae9c6921fb48 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:last_changed 2023-11-24T06:17:49.634155+0000 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:created 2023-11-24T06:17:49.634155+0000 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:min_mon_release 16 (pacific) 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:election_strategy: 1 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:1: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:2: [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] mon.c 2023-11-24T06:17:49.636 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-24T06:17:49.638 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T06:17:49.653 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID be2488f4-68a7-42cc-93d1-ae9c6921fb48... 2023-11-24T06:17:49.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-24T06:17:49.696 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-24T06:17:49.739 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2023-11-24T06:17:49.739 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2023-11-24T06:17:49.739 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T06:17:49.739 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2023-11-24T06:17:49.739 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.739 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2023-11-24T06:17:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2023-11-24T06:17:49.741 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.89,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]" 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on skipped message = False 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = be2488f4-68a7-42cc-93d1-ae9c6921fb48 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2023-11-24T06:17:49.743 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2023-11-24T06:17:49.744 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2023-11-24T06:17:49.745 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = bitmap 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T06:17:49.746 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2023-11-24T06:17:49.747 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: debug finisher = 20 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2023-11-24T06:17:49.748 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.749 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: mon op complaint time = 120 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T06:17:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: client mount timeout = 600 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = "" 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2023-11-24T06:17:49.751 INFO:teuthology.orchestra.run.smithi089.stdout:[mds] 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds = 20 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds balancer = 20 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug frag = True 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug scatterstat = True 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: mds op complaint time = 180 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: mds verify scatter = True 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2023-11-24T06:17:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2023-11-24T06:17:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2023-11-24T06:17:49.754 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: chdir = "" 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: auth supported = cephx 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: filestore xattr use omap = true 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: mon clock drift allowed = 1.000 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: osd crush chooseleaf type = 0 2023-11-24T06:17:49.755 INFO:teuthology.orchestra.run.smithi123.stdout: auth debug = true 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on old message = true 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on bug = true 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: mon pg warn max object skew = 0 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default size = 2 2023-11-24T06:17:49.756 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow primary affinity = true 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow pg remap = true 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on legacy crush tunables = false 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on crush straw calc version zero = false 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on no sortbitwise = false 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on osd down out interval zero = false 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on too few osds = false 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T06:17:49.757 INFO:teuthology.orchestra.run.smithi123.stdout: mon_allow_pool_size_one = true 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: osd default data pool replay window = 5 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: mon allow pool delete = true 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: mon cluster log file level = debug 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: debug asserts on shutdown = true 2023-11-24T06:17:49.758 INFO:teuthology.orchestra.run.smithi123.stdout: mon health detail to clog = false 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: mon host = "172.21.15.89,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]" 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on skipped message = False 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = be2488f4-68a7-42cc-93d1-ae9c6921fb48 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout:[osd] 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: osd journal size = 100 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub load threshold = 5.0 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub max interval = 600 2023-11-24T06:17:49.759 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock profile = high_recovery_ops 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: osd recover clone overlap = true 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: osd recovery max chunk = 1048576 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug shutdown = true 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug op order = true 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify stray on activate = true 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug trim objects = true 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.760 INFO:teuthology.orchestra.run.smithi123.stdout: osd open classes on start = true 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug pg log writeout = true 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: osd deep scrub update digest min age = 30 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: osd map max advance = 10 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: journal zero on create = true 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: filestore ondisk finisher threads = 3 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: filestore apply finisher threads = 3 2023-11-24T06:17:49.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.762 INFO:teuthology.orchestra.run.smithi123.stdout: bdev debug aio = true 2023-11-24T06:17:49.762 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug misdirected ops = true 2023-11-24T06:17:49.762 INFO:teuthology.orchestra.run.smithi123.stdout: bdev async discard = True 2023-11-24T06:17:49.762 INFO:teuthology.orchestra.run.smithi123.stdout: bdev enable discard = True 2023-11-24T06:17:49.762 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore allocator = bitmap 2023-11-24T06:17:49.762 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore block size = 96636764160 2023-11-24T06:17:49.762 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore fsck on mount = True 2023-11-24T06:17:49.762 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluefs = 1/20 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluestore = 1/20 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: debug osd = 20 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: debug rocksdb = 4/10 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd full ratio = 0.9 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd nearfull ratio = 0.8 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: osd failsafe full ratio = 0.95 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: osd objectstore = bluestore 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: osd op complaint time = 180 2023-11-24T06:17:49.763 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr] 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgr = 20 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: mgr/telemetry/nag = false 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: debug finisher = 20 2023-11-24T06:17:49.764 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout:[mon] 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: debug paxos = 20 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: mon data avail warn = 5 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: mon mgr mkfs grace = 240 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd reporter subtree level = osd 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd prime pg temp = true 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2023-11-24T06:17:49.765 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: # 1m isn't quite enough 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: mon_down_mkfs_grace = 2m 2023-11-24T06:17:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_filestore_osds = false 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: mon op complaint time = 120 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout:[client] 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: rgw cache enabled = true 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable ops log = true 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable usage log = true 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: client mount timeout = 600 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2023-11-24T06:17:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = "" 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: rados mon op timeout = 900 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: rados osd op timeout = 900 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.a] 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.b] 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.c] 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout:[mds] 2023-11-24T06:17:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: debug mds = 20 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: debug mds balancer = 20 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: mds debug frag = True 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: mds debug scatterstat = True 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: mds op complaint time = 180 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: mds verify scatter = True 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: osd op complaint time = 180 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: rados mon op timeout = 900 2023-11-24T06:17:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: rados osd op timeout = 900 2023-11-24T06:17:49.769 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-24T06:17:49.770 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-24T06:17:49.852 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-24T06:17:49.852 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:49.852 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-24T06:17:49.868 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:49.868 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-24T06:17:49.926 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:17:49.926 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:49.926 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-24T06:17:49.927 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T06:17:50.002 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:17:50.002 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:17:50.058 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:17:50.059 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:17:50.059 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-24T06:17:50.059 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T06:17:50.096 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:17:50.097 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:17:50.153 INFO:tasks.ceph:Setting up mon nodes... 2023-11-24T06:17:50.153 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-24T06:17:50.153 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-24T06:17:50.206 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-24T06:17:50.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-24T06:17:50.270 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-24T06:17:50.273 INFO:tasks.ceph:Setting up mds nodes... 2023-11-24T06:17:50.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-24T06:17:50.327 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-24T06:17:50.329 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-24T06:17:50.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-24T06:17:50.448 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-24T06:17:50.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-24T06:17:50.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-24T06:17:50.529 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-24T06:17:50.531 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-24T06:17:50.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-24T06:17:50.650 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-24T06:17:50.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-24T06:17:50.679 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-24T06:17:50.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-24T06:17:50.721 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-24T06:17:50.768 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-11-24T06:17:50.851 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-11-24T06:17:50.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-24T06:17:50.909 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-24T06:17:50.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-11-24T06:17:50.962 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-11-24T06:17:50.975 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-24T06:17:50.976 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-24T06:17:50.976 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-24T06:17:51.002 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:17:51.002 INFO:tasks.ceph:role: osd.0 2023-11-24T06:17:51.003 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:17:51.003 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:17:51.076 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:17:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:17:51.509 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2023-11-24T06:17:51.512 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2023-11-24T06:17:51.512 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-24T06:17:51.672 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-24T06:17:51.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-24T06:17:51.777 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:17:51.777 INFO:tasks.ceph:role: osd.1 2023-11-24T06:17:51.777 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:17:51.777 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-24T06:17:51.851 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:17:51.851 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:17:51.851 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:17:51.852 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:17:51.852 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:17:51.852 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:17:51.852 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:17:51.852 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:17:51.852 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:17:51.852 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:17:52.276 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2023-11-24T06:17:52.277 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2023-11-24T06:17:52.278 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-24T06:17:52.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-24T06:17:52.419 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-24T06:17:52.488 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:17:52.488 INFO:tasks.ceph:role: osd.2 2023-11-24T06:17:52.488 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:17:52.488 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-24T06:17:52.564 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:17:52.564 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:17:52.565 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:17:52.565 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:17:52.565 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:17:52.565 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:17:52.565 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:17:52.565 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:17:52.565 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:17:52.565 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:17:53.017 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2023-11-24T06:17:53.019 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2023-11-24T06:17:53.019 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-24T06:17:53.176 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-24T06:17:53.212 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-24T06:17:53.280 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:17:53.280 INFO:tasks.ceph:role: osd.3 2023-11-24T06:17:53.280 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:17:53.281 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-24T06:17:53.356 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:17:53.356 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:17:53.356 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:17:53.357 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:17:53.357 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:17:53.357 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:17:53.357 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:17:53.357 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:17:53.358 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:17:53.358 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:17:53.794 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2023-11-24T06:17:53.796 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2023-11-24T06:17:53.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-24T06:17:53.973 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-24T06:17:54.012 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:17:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:54.119+0000 7f539b9ed640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-24T06:17:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:54.119+0000 7f539b9ed640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-24T06:17:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:54.119+0000 7f539b9ed640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T06:17:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:54.119+0000 7f539b9ed640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T06:17:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:54.119+0000 7f539b9ed640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T06:17:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:54.120+0000 7f539b9ed640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-24T06:17:56.982 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-24T06:17:57.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:17:57.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:57.104+0000 7fc44e1e8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-24T06:17:57.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:57.104+0000 7fc44e1e8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-24T06:17:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:57.104+0000 7fc44e1e8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T06:17:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:57.104+0000 7fc44e1e8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T06:17:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:57.104+0000 7fc44e1e8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T06:17:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:17:57.105+0000 7fc44e1e8640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-24T06:17:59.973 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-24T06:18:00.002 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:18:00.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:00.089+0000 7fa4e9006640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-24T06:18:00.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:00.089+0000 7fa4e9006640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-24T06:18:00.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:00.089+0000 7fa4e9006640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T06:18:00.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:00.089+0000 7fa4e9006640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T06:18:00.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:00.089+0000 7fa4e9006640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T06:18:00.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:00.090+0000 7fa4e9006640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-24T06:18:02.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-24T06:18:02.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:18:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:03.086+0000 7f34fd60f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-24T06:18:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:03.086+0000 7f34fd60f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-24T06:18:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:03.086+0000 7f34fd60f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T06:18:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:03.086+0000 7f34fd60f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T06:18:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:03.086+0000 7f34fd60f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T06:18:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:03.087+0000 7f34fd60f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-24T06:18:05.957 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-24T06:18:05.989 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-24T06:18:06.023 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:18:06.023 INFO:tasks.ceph:role: osd.4 2023-11-24T06:18:06.023 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:18:06.024 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-24T06:18:06.104 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:18:06.104 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:18:06.104 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:18:06.104 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:18:06.104 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:18:06.105 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:18:06.105 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:18:06.105 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:18:06.105 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:18:06.105 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:18:06.540 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-11-24T06:18:06.542 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-11-24T06:18:06.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-24T06:18:06.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-24T06:18:06.747 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-24T06:18:06.816 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:18:06.817 INFO:tasks.ceph:role: osd.5 2023-11-24T06:18:06.817 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:18:06.817 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-24T06:18:06.893 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:18:06.893 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:18:06.894 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:18:06.894 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:18:06.894 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:18:06.894 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:18:06.894 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:18:06.894 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:18:06.894 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:18:06.894 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:18:07.325 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-11-24T06:18:07.327 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-11-24T06:18:07.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-24T06:18:07.431 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-24T06:18:07.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-24T06:18:07.529 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:18:07.529 INFO:tasks.ceph:role: osd.6 2023-11-24T06:18:07.529 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:18:07.529 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-24T06:18:07.603 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:18:07.603 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:18:07.603 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:18:07.604 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:18:07.604 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:18:07.604 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:18:07.604 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:18:07.604 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:18:07.604 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:18:07.604 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:18:08.050 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-11-24T06:18:08.051 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-11-24T06:18:08.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-24T06:18:08.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-24T06:18:08.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-24T06:18:08.313 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:18:08.314 INFO:tasks.ceph:role: osd.7 2023-11-24T06:18:08.314 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:18:08.314 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-24T06:18:08.388 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:18:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:18:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:18:08.388 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:18:08.388 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:18:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:18:08.389 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:18:08.389 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:18:08.389 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:18:08.389 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:18:08.837 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-11-24T06:18:08.839 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-11-24T06:18:08.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-24T06:18:09.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-24T06:18:09.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:18:09.156 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:09.153+0000 7f888d2b6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-24T06:18:09.156 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:09.153+0000 7f888d2b6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-24T06:18:09.156 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:09.153+0000 7f888d2b6640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T06:18:09.156 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:09.153+0000 7f888d2b6640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T06:18:09.157 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:09.153+0000 7f888d2b6640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T06:18:09.157 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:09.154+0000 7f888d2b6640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-24T06:18:12.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-24T06:18:12.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:18:12.150 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:12.148+0000 7f4909ce5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-24T06:18:12.151 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:12.148+0000 7f4909ce5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-24T06:18:12.151 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:12.148+0000 7f4909ce5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T06:18:12.151 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:12.148+0000 7f4909ce5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T06:18:12.151 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:12.148+0000 7f4909ce5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T06:18:12.151 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:12.149+0000 7f4909ce5640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-24T06:18:15.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-24T06:18:15.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:18:15.142 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:15.140+0000 7f20cabe4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-24T06:18:15.142 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:15.140+0000 7f20cabe4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-24T06:18:15.143 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:15.140+0000 7f20cabe4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T06:18:15.143 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:15.140+0000 7f20cabe4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T06:18:15.143 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:15.140+0000 7f20cabe4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T06:18:15.143 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:15.141+0000 7f20cabe4640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-24T06:18:17.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:18:17.722 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:18:18.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-24T06:18:18.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:18:18.142 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:18.140+0000 7ffa77e81640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-24T06:18:18.142 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:18.140+0000 7ffa77e81640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-24T06:18:18.142 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:18.140+0000 7ffa77e81640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T06:18:18.142 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:18.140+0000 7ffa77e81640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T06:18:18.142 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:18.140+0000 7ffa77e81640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T06:18:18.143 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:18.140+0000 7ffa77e81640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-24T06:18:21.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-24T06:18:21.047 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-24T06:18:21.047 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-24T06:18:21.079 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.079 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-24T06:18:21.151 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-24T06:18:21.221 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.221 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-24T06:18:21.289 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.289 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-24T06:18:21.358 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.358 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-24T06:18:21.426 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.440 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-24T06:18:21.494 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:21.494 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-24T06:18:21.527 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:21.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-24T06:18:21.599 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:21.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-24T06:18:21.670 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:21.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-24T06:18:21.743 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:21.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-24T06:18:21.814 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:21.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-24T06:18:21.885 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:21.885 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-24T06:18:21.955 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.955 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-24T06:18:21.975 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:21.975 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-11-24T06:18:22.035 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:22.035 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-24T06:18:22.053 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:22.054 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-11-24T06:18:22.112 INFO:tasks.ceph:Adding keys to all mons... 2023-11-24T06:18:22.112 DEBUG:teuthology.orchestra.run.smithi089:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-24T06:18:22.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-24T06:18:22.155 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.y] 2023-11-24T06:18:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlsrk+DBAAdgV15cObrTUfehK6bUY8DA== 2023-11-24T06:18:22.155 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlZPhnExAAdSzrCYJj/BqJINrd+mxnog== 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlrHqfGhAA7hKW48g57uUyHOiS4gQkOQ== 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.0] 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQASQGBlLUQyBxAApB8r9XPrC9s6zsd2nPQFoQ== 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.1] 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAVQGBlFgtPBhAAvSzzgIHzad8zv8vS3fcFQg== 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.2] 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAYQGBlNW1sBRAAtOtIYUOAa63AiLIS9EYCFA== 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.3] 2023-11-24T06:18:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAbQGBlZOIsBRAATWs4ayExofw/56n0ixjgDA== 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.x] 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlj+EGEBAAFs6HBnMAVB9ZLdn7WrWpzQ== 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlAvtyHxAAiLxnKUSDAvkxFP6LHLBkSQ== 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBl4PG2JhAAg8KlX9jNiQ/Ajvrm71BM8g== 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.4] 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAhQGBlGkIpCRAAWpMWRhHqDSEuyM+DAQLP8Q== 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.5] 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAkQGBli6LfCBAAH5LBKEpbvU48rAyWnyH0Zg== 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.6] 2023-11-24T06:18:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAnQGBl6qpjCBAASFc7C3OIr9G6lSXCNyLk8A== 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.7] 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAqQGBlxmJdCBAANGRCESck0mrD1gVuToRRaw== 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlwIfyKhAAMZcwOQGAm+XcH+mUKcTiUg== 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout:[client.2] 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBluCSzMhAA1hxy9u3cvwTZLG/Xg3kQWQ== 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout:[client.1] 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlxNQDNhAApU+JeKCeMTDo61+wi3ynbw== 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout:[client.3] 2023-11-24T06:18:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlin9NORAAkdx6VeU4rwbgNfVPwwUXFg== 2023-11-24T06:18:22.180 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.y] 2023-11-24T06:18:22.180 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBlsrk+DBAAdgV15cObrTUfehK6bUY8DA== 2023-11-24T06:18:22.180 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2023-11-24T06:18:22.180 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBlZPhnExAAdSzrCYJj/BqJINrd+mxnog== 2023-11-24T06:18:22.180 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2023-11-24T06:18:22.180 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBlrHqfGhAA7hKW48g57uUyHOiS4gQkOQ== 2023-11-24T06:18:22.180 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.0] 2023-11-24T06:18:22.180 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQASQGBlLUQyBxAApB8r9XPrC9s6zsd2nPQFoQ== 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.1] 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAVQGBlFgtPBhAAvSzzgIHzad8zv8vS3fcFQg== 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.2] 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAYQGBlNW1sBRAAtOtIYUOAa63AiLIS9EYCFA== 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.3] 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAbQGBlZOIsBRAATWs4ayExofw/56n0ixjgDA== 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.x] 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBlj+EGEBAAFs6HBnMAVB9ZLdn7WrWpzQ== 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2023-11-24T06:18:22.181 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBlAvtyHxAAiLxnKUSDAvkxFP6LHLBkSQ== 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBl4PG2JhAAg8KlX9jNiQ/Ajvrm71BM8g== 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.4] 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAhQGBlGkIpCRAAWpMWRhHqDSEuyM+DAQLP8Q== 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.5] 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAkQGBli6LfCBAAH5LBKEpbvU48rAyWnyH0Zg== 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.6] 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAnQGBl6qpjCBAASFc7C3OIr9G6lSXCNyLk8A== 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.7] 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAqQGBlxmJdCBAANGRCESck0mrD1gVuToRRaw== 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2023-11-24T06:18:22.182 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBlwIfyKhAAMZcwOQGAm+XcH+mUKcTiUg== 2023-11-24T06:18:22.183 INFO:teuthology.orchestra.run.smithi123.stdout:[client.2] 2023-11-24T06:18:22.183 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBluCSzMhAA1hxy9u3cvwTZLG/Xg3kQWQ== 2023-11-24T06:18:22.183 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2023-11-24T06:18:22.183 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBlxNQDNhAApU+JeKCeMTDo61+wi3ynbw== 2023-11-24T06:18:22.183 INFO:teuthology.orchestra.run.smithi123.stdout:[client.3] 2023-11-24T06:18:22.183 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAOQGBlin9NORAAkdx6VeU4rwbgNfVPwwUXFg== 2023-11-24T06:18:22.184 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T06:18:22.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T06:18:22.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:18:22.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:18:22.340 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:18:22.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:18:22.398 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.453 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.508 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.618 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T06:18:22.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T06:18:22.672 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:18:22.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:18:22.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:18:22.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:18:22.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.847 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.848 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.904 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.957 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:22.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:18:23.013 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:18:23.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:18:23.071 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:18:23.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:18:23.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:18:23.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:18:23.188 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:18:23.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:18:23.246 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-24T06:18:23.246 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-24T06:18:23.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T06:18:23.373 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-24T06:18:23.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-24T06:18:23.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T06:18:23.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-24T06:18:23.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-24T06:18:23.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T06:18:23.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-24T06:18:23.742 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:18:23.745 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:18:23.798 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-24T06:18:23.799 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-24T06:18:23.799 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-24T06:18:23.801 INFO:tasks.ceph.mon.a:Started 2023-11-24T06:18:23.801 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-24T06:18:23.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-24T06:18:23.841 INFO:tasks.ceph.mon.b:Started 2023-11-24T06:18:23.841 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-24T06:18:23.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-24T06:18:23.843 INFO:tasks.ceph.mon.c:Started 2023-11-24T06:18:23.843 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-24T06:18:23.843 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-24T06:18:23.843 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-24T06:18:23.845 INFO:tasks.ceph.mgr.y:Started 2023-11-24T06:18:23.846 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-24T06:18:23.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-24T06:18:23.847 INFO:tasks.ceph.mgr.x:Started 2023-11-24T06:18:23.848 INFO:tasks.ceph:Setting crush tunables to default 2023-11-24T06:18:23.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd crush tunables default 2023-11-24T06:18:23.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.968+0000 7fc8f1b6d700 1 Processor -- start 2023-11-24T06:18:23.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.968+0000 7fc8f1b6d700 1 -- start start 2023-11-24T06:18:23.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8f1b6d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 0x7fc8ec0ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:23.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8f1b6d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8f1b6d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8f1b6d700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec15c330 con 0x7fc8ec0ce830 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8f1b6d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec0d0140 con 0x7fc8ec15bf50 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8f1b6d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0d03e0 con 0x7fc8ec0cf1f0 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eaffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eaffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34342/0 (socket says 172.21.15.89:34342) 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eaffd700 1 -- 172.21.15.89:0/3636813926 learned_addr learned my addr 172.21.15.89:0/3636813926 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eaffd700 1 -- 172.21.15.89:0/3636813926 >> v2:172.21.15.89:3300/0 conn(0x7fc8ec0ce830 msgr2=0x7fc8ec0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eaffd700 1 -- 172.21.15.89:0/3636813926 >> v2:172.21.15.89:3300/0 conn(0x7fc8ec0ce830 msgr2=0x7fc8ec0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.89:3300/0 conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T06:18:23.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.89:3300/0 conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T06:18:23.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8ebfff700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:23.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eb7fe700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 0x7fc8ec0ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:23.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8ebfff700 1 -- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 msgr2=0x7fc8ec0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-24T06:18:23.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8ebfff700 1 -- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 msgr2=0x7fc8ec0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T06:18:23.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8ebfff700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T06:18:23.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8ebfff700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0c5c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T06:18:23.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eb7fe700 1 -- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 msgr2=0x7fc8ec0ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-24T06:18:23.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eb7fe700 1 -- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 msgr2=0x7fc8ec0ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T06:18:23.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eb7fe700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 0x7fc8ec0ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T06:18:23.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:23.969+0000 7fc8eb7fe700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 0x7fc8ec0ce2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T06:18:24.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.89:3300/0 conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8ebfff700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eb7fe700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 0x7fc8ec0ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eaffd700 1 -- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 msgr2=0x7fc8ec0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eaffd700 1 -- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 msgr2=0x7fc8ec0ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 0x7fc8ec0ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eaffd700 1 -- 172.21.15.89:0/3636813926 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec0d06c0 con 0x7fc8ec0ce830 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eb7fe700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 0x7fc8ec0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0715e0 tx=0x7fc8e0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c76e24b302b822e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:24.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8ea7fc700 1 -- 172.21.15.89:0/3636813926 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0004230 con 0x7fc8ec0ce830 2023-11-24T06:18:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8ea7fc700 1 -- 172.21.15.89:0/3636813926 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e0004390 con 0x7fc8ec0ce830 2023-11-24T06:18:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.170+0000 7fc8ea7fc700 1 -- 172.21.15.89:0/3636813926 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e00036a0 con 0x7fc8ec0ce830 2023-11-24T06:18:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/3636813926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0ce830 msgr2=0x7fc8ec0cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0715e0 tx=0x7fc8e0009c30 comp rx=0 tx=0).stop 2023-11-24T06:18:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/3636813926 shutdown_connections 2023-11-24T06:18:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 --2- 172.21.15.89:0/3636813926 >> v2:172.21.15.123:3300/0 conn(0x7fc8ec15bf50 0x7fc8ec0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 --2- 172.21.15.89:0/3636813926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0ce830 0x7fc8ec0cecb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/3636813926 >> 172.21.15.89:0/3636813926 conn(0x7fc8ec0b3780 msgr2=0x7fc8ec0b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/3636813926 shutdown_connections 2023-11-24T06:18:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.171+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/3636813926 wait complete. 2023-11-24T06:18:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.173+0000 7fc8f1b6d700 1 Processor -- start 2023-11-24T06:18:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.173+0000 7fc8f1b6d700 1 -- start start 2023-11-24T06:18:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8f1b6d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ec0ce830 0x7fc8ec0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8f1b6d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8f1b6d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec15bf50 0x7fc8ec0b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8f1b6d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0caca0 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8f1b6d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0caf80 con 0x7fc8ec0ce830 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8f1b6d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0cb260 con 0x7fc8ec15bf50 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0b64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34372/0 (socket says 172.21.15.89:34372) 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 -- 172.21.15.89:0/111234849 learned_addr learned my addr 172.21.15.89:0/111234849 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eb7fe700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ec0ce830 0x7fc8ec0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8ebfff700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec15bf50 0x7fc8ec0b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 -- 172.21.15.89:0/111234849 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec15bf50 msgr2=0x7fc8ec0b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec15bf50 0x7fc8ec0b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 -- 172.21.15.89:0/111234849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ec0ce830 msgr2=0x7fc8ec0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ec0ce830 0x7fc8ec0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 -- 172.21.15.89:0/111234849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e00095b0 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8eaffd700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0b64a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0015040 tx=0x7fc8e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8d3fff700 1 -- 172.21.15.89:0/111234849 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0013be0 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8d3fff700 1 -- 172.21.15.89:0/111234849 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e0013d40 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec0cb540 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8d3fff700 1 -- 172.21.15.89:0/111234849 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0025650 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.175+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec154390 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.176+0000 7fc8d3fff700 1 -- 172.21.15.89:0/111234849 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc8e0025ab0 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.176+0000 7fc8d3fff700 1 -- 172.21.15.89:0/111234849 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc8e002fa60 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.176+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec154580 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.179+0000 7fc8d3fff700 1 -- 172.21.15.89:0/111234849 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc8e002a070 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.255+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc8ec068fb0 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.256+0000 7fc8d3fff700 1 -- 172.21.15.89:0/111234849 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc8e000c070 con 0x7fc8ec0cf1f0 2023-11-24T06:18:24.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.257+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0cf1f0 msgr2=0x7fc8ec0b64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.257+0000 7fc8f1b6d700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0b64a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0015040 tx=0x7fc8e00047c0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.258+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 shutdown_connections 2023-11-24T06:18:24.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.258+0000 7fc8f1b6d700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8ec15bf50 0x7fc8ec0b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.258+0000 7fc8f1b6d700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ec0ce830 0x7fc8ec0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.258+0000 7fc8f1b6d700 1 --2- 172.21.15.89:0/111234849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ec0cf1f0 0x7fc8ec0b64a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.258+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 >> 172.21.15.89:0/111234849 conn(0x7fc8ec0b3780 msgr2=0x7fc8ec06f490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:24.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.258+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 shutdown_connections 2023-11-24T06:18:24.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:24.258+0000 7fc8f1b6d700 1 -- 172.21.15.89:0/111234849 wait complete. 2023-11-24T06:18:24.262 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2023-11-24T06:18:24.273 INFO:tasks.ceph:check_enable_crimson: False 2023-11-24T06:18:24.273 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-24T06:18:24.273 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:24.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-24T06:18:24.281 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:24.278+0000 7f9c13b1a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-24T06:18:24.284 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:24.282+0000 7fab736cf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-24T06:18:24.299 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:24.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-24T06:18:24.345 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:24.343+0000 7f9c13b1a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-24T06:18:24.352 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:24.350+0000 7fab736cf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-24T06:18:24.364 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:24.364 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-24T06:18:24.421 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:24.418+0000 7f9c13b1a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-24T06:18:24.430 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:18:24.430 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-24T06:18:24.433 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:24.430+0000 7fab736cf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-24T06:18:24.458 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:24.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-24T06:18:24.487 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:24.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-24T06:18:24.504 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:24.501+0000 7f9c13b1a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-24T06:18:24.516 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:24.513+0000 7fab736cf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-24T06:18:24.531 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:24.531 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-24T06:18:24.599 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:18:24.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-24T06:18:24.657 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:24.654+0000 7f9c13b1a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-24T06:18:24.658 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:24.656+0000 7fab736cf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-24T06:18:24.664 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new a3933529-39bc-4e58-a042-b9028c56f953 0 2023-11-24T06:18:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.823+0000 7ffa3125b700 1 Processor -- start 2023-11-24T06:18:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.823+0000 7ffa3125b700 1 -- start start 2023-11-24T06:18:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.824+0000 7ffa3125b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffa2c15ada0 0x7ffa2c06f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.824+0000 7ffa3125b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c06d3c0 0x7ffa2c06d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.824+0000 7ffa3125b700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0cccd0 con 0x7ffa2c0afb40 2023-11-24T06:18:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.824+0000 7ffa3125b700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cd2e0 con 0x7ffa2c15ada0 2023-11-24T06:18:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.824+0000 7ffa3125b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cd5c0 con 0x7ffa2c06d3c0 2023-11-24T06:18:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.824+0000 7ffa2b59e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c06d3c0 0x7ffa2c06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.824+0000 7ffa2b59e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c06d3c0 0x7ffa2c06d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42880/0 (socket says 172.21.15.123:42880) 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.824+0000 7ffa2b59e700 1 -- 172.21.15.123:0/3496186995 learned_addr learned my addr 172.21.15.123:0/3496186995 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa2ad9d700 1 --2- 172.21.15.123:0/3496186995 >> v2:172.21.15.89:3300/0 conn(0x7ffa2c15ada0 0x7ffa2c06f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa2b59e700 1 -- 172.21.15.123:0/3496186995 >> v2:172.21.15.89:3300/0 conn(0x7ffa2c15ada0 msgr2=0x7ffa2c06f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa2b59e700 1 --2- 172.21.15.123:0/3496186995 >> v2:172.21.15.89:3300/0 conn(0x7ffa2c15ada0 0x7ffa2c06f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa2b59e700 1 -- 172.21.15.123:0/3496186995 >> v1:172.21.15.89:6789/0 conn(0x7ffa2c0afb40 legacy=0x7ffa2c0cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa2b59e700 1 -- 172.21.15.123:0/3496186995 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c06fd50 con 0x7ffa2c06d3c0 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa2b59e700 1 --2- 172.21.15.123:0/3496186995 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c06d3c0 0x7ffa2c06d850 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009e50 tx=0x7ffa2400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a83b0cd058a2dde0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa29d9b700 1 -- 172.21.15.123:0/3496186995 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2400e070 con 0x7ffa2c06d3c0 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa29d9b700 1 -- 172.21.15.123:0/3496186995 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2400b910 con 0x7ffa2c06d3c0 2023-11-24T06:18:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.825+0000 7ffa29d9b700 1 -- 172.21.15.123:0/3496186995 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2400bcc0 con 0x7ffa2c06d3c0 2023-11-24T06:18:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.826+0000 7ffa3125b700 1 -- 172.21.15.123:0/3496186995 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c06d3c0 msgr2=0x7ffa2c06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.826+0000 7ffa3125b700 1 --2- 172.21.15.123:0/3496186995 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c06d3c0 0x7ffa2c06d850 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009e50 tx=0x7ffa2400b330 comp rx=0 tx=0).stop 2023-11-24T06:18:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.826+0000 7ffa3125b700 1 -- 172.21.15.123:0/3496186995 shutdown_connections 2023-11-24T06:18:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.826+0000 7ffa3125b700 1 --2- 172.21.15.123:0/3496186995 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c06d3c0 0x7ffa2c06d850 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.826+0000 7ffa3125b700 1 --2- 172.21.15.123:0/3496186995 >> v2:172.21.15.89:3300/0 conn(0x7ffa2c15ada0 0x7ffa2c06f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.826+0000 7ffa3125b700 1 -- 172.21.15.123:0/3496186995 >> 172.21.15.123:0/3496186995 conn(0x7ffa2c0b2050 msgr2=0x7ffa2c0b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.826+0000 7ffa3125b700 1 -- 172.21.15.123:0/3496186995 shutdown_connections 2023-11-24T06:18:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.826+0000 7ffa3125b700 1 -- 172.21.15.123:0/3496186995 wait complete. 2023-11-24T06:18:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.828+0000 7ffa3125b700 1 Processor -- start 2023-11-24T06:18:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.828+0000 7ffa3125b700 1 -- start start 2023-11-24T06:18:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.829+0000 7ffa3125b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa2c06d3c0 0x7ffa2c15d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.829+0000 7ffa3125b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c0afb40 0x7ffa2c0b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.829+0000 7ffa3125b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c15ada0 0x7ffa2c1f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.829+0000 7ffa3125b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c1f3b90 con 0x7ffa2c15ada0 2023-11-24T06:18:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.829+0000 7ffa3125b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c15de80 con 0x7ffa2c06d3c0 2023-11-24T06:18:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.829+0000 7ffa3125b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c1f3e30 con 0x7ffa2c0afb40 2023-11-24T06:18:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.829+0000 7ffa2ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa2c06d3c0 0x7ffa2c15d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.829+0000 7ffa2a59c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c0afb40 0x7ffa2c0b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa2c06d3c0 0x7ffa2c15d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58334/0 (socket says 172.21.15.123:58334) 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2ad9d700 1 -- 172.21.15.123:0/1120605866 learned_addr learned my addr 172.21.15.123:0/1120605866 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2b59e700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c15ada0 0x7ffa2c1f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2b59e700 1 -- 172.21.15.123:0/1120605866 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c0afb40 msgr2=0x7ffa2c0b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2b59e700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c0afb40 0x7ffa2c0b36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2b59e700 1 -- 172.21.15.123:0/1120605866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa2c06d3c0 msgr2=0x7ffa2c15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2b59e700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa2c06d3c0 0x7ffa2c15d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2b59e700 1 -- 172.21.15.123:0/1120605866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa24009b30 con 0x7ffa2c15ada0 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2a59c700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c0afb40 0x7ffa2c0b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2ad9d700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa2c06d3c0 0x7ffa2c15d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:24.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa2b59e700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c15ada0 0x7ffa2c1f0a60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa24015000 tx=0x7ffa24004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:24.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa137fe700 1 -- 172.21.15.123:0/1120605866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24004800 con 0x7ffa2c15ada0 2023-11-24T06:18:24.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa137fe700 1 -- 172.21.15.123:0/1120605866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24004960 con 0x7ffa2c15ada0 2023-11-24T06:18:24.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c1f4110 con 0x7ffa2c15ada0 2023-11-24T06:18:24.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa137fe700 1 -- 172.21.15.123:0/1120605866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2401c680 con 0x7ffa2c15ada0 2023-11-24T06:18:24.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c15e160 con 0x7ffa2c15ada0 2023-11-24T06:18:24.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa137fe700 1 -- 172.21.15.123:0/1120605866 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ffa2401cae0 con 0x7ffa2c15ada0 2023-11-24T06:18:24.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.830+0000 7ffa137fe700 1 -- 172.21.15.123:0/1120605866 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ffa24025be0 con 0x7ffa2c15ada0 2023-11-24T06:18:24.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.831+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c068d90 con 0x7ffa2c15ada0 2023-11-24T06:18:24.837 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.833+0000 7ffa137fe700 1 -- 172.21.15.123:0/1120605866 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ffa24020070 con 0x7ffa2c15ada0 2023-11-24T06:18:24.920 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.918+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a3933529-39bc-4e58-a042-b9028c56f953", "id": 0} v 0) v1 -- 0x7ffa2c1f3f70 con 0x7ffa2c15ada0 2023-11-24T06:18:24.922 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.919+0000 7ffa137fe700 1 -- 172.21.15.123:0/1120605866 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a3933529-39bc-4e58-a042-b9028c56f953", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffa2c1f3f70 con 0x7ffa2c15ada0 2023-11-24T06:18:24.923 INFO:teuthology.orchestra.run.smithi123.stdout:0 2023-11-24T06:18:24.924 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c15ada0 msgr2=0x7ffa2c1f0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:24.924 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c15ada0 0x7ffa2c1f0a60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa24015000 tx=0x7ffa24004ad0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.924 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 shutdown_connections 2023-11-24T06:18:24.925 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c15ada0 0x7ffa2c1f0a60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.925 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa2c06d3c0 0x7ffa2c15d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.925 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 --2- 172.21.15.123:0/1120605866 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa2c0afb40 0x7ffa2c0b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:24.925 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 >> 172.21.15.123:0/1120605866 conn(0x7ffa2c0b2050 msgr2=0x7ffa2c0b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:24.925 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 shutdown_connections 2023-11-24T06:18:24.925 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:24.920+0000 7ffa3125b700 1 -- 172.21.15.123:0/1120605866 wait complete. 2023-11-24T06:18:24.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 97e78a03-85d2-414c-8968-f999dd8e1126 1 2023-11-24T06:18:25.052 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:25.049+0000 7fab736cf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-24T06:18:25.055 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.053+0000 7fd31f676700 1 Processor -- start 2023-11-24T06:18:25.055 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.053+0000 7fd31f676700 1 -- start start 2023-11-24T06:18:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31f676700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31806bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31f676700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd3180ce430 0x7fd3180ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31f676700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31806ba50 con 0x7fd31815e080 2023-11-24T06:18:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31f676700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3180d1200 con 0x7fd3180ce430 2023-11-24T06:18:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31f676700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180cee20 con 0x7fd3180b4310 2023-11-24T06:18:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31cc11700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31806bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31d412700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd31815e080 0x7fd3180d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:38566/0 (socket says 172.21.15.123:38566) 2023-11-24T06:18:25.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31cc11700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31806bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42890/0 (socket says 172.21.15.123:42890) 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31d412700 1 -- 172.21.15.123:0/3631722677 learned_addr learned my addr 172.21.15.123:0/3631722677 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31dc13700 1 --2- 172.21.15.123:0/3631722677 >> v2:172.21.15.89:3300/0 conn(0x7fd3180ce430 0x7fd3180ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31cc11700 1 -- 172.21.15.123:0/3631722677 >> v2:172.21.15.89:3300/0 conn(0x7fd3180ce430 msgr2=0x7fd3180ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31cc11700 1 --2- 172.21.15.123:0/3631722677 >> v2:172.21.15.89:3300/0 conn(0x7fd3180ce430 0x7fd3180ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31cc11700 1 -- 172.21.15.123:0/3631722677 >> v1:172.21.15.123:6789/0 conn(0x7fd31815e080 legacy=0x7fd3180d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd30ffff700 1 -- 172.21.15.123:0/3631722677 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547775762 0 0) 0x7fd31806ba50 con 0x7fd31815e080 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31cc11700 1 -- 172.21.15.123:0/3631722677 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31806b120 con 0x7fd3180b4310 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd31cc11700 1 --2- 172.21.15.123:0/3631722677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31806bee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd308009840 tx=0x7fd308009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cf462b8ed2993d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd30ffff700 1 -- 172.21.15.123:0/3631722677 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308011830 con 0x7fd3180b4310 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd30ffff700 1 -- 172.21.15.123:0/3631722677 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd308011990 con 0x7fd3180b4310 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.054+0000 7fd30ffff700 1 -- 172.21.15.123:0/3631722677 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308011d20 con 0x7fd3180b4310 2023-11-24T06:18:25.059 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.055+0000 7fd31f676700 1 -- 172.21.15.123:0/3631722677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 msgr2=0x7fd31806bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.060 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.055+0000 7fd31f676700 1 --2- 172.21.15.123:0/3631722677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31806bee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd308009840 tx=0x7fd308009b50 comp rx=0 tx=0).stop 2023-11-24T06:18:25.060 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.055+0000 7fd31f676700 1 -- 172.21.15.123:0/3631722677 shutdown_connections 2023-11-24T06:18:25.060 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.055+0000 7fd31f676700 1 --2- 172.21.15.123:0/3631722677 >> v2:172.21.15.89:3300/0 conn(0x7fd3180ce430 0x7fd3180ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.060 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.055+0000 7fd31f676700 1 --2- 172.21.15.123:0/3631722677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31806bee0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.060 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.055+0000 7fd31f676700 1 -- 172.21.15.123:0/3631722677 >> 172.21.15.123:0/3631722677 conn(0x7fd3180b3780 msgr2=0x7fd3180b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:25.060 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.055+0000 7fd31f676700 1 -- 172.21.15.123:0/3631722677 shutdown_connections 2023-11-24T06:18:25.060 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.055+0000 7fd31f676700 1 -- 172.21.15.123:0/3631722677 wait complete. 2023-11-24T06:18:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.056+0000 7fd31f676700 1 Processor -- start 2023-11-24T06:18:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.056+0000 7fd31f676700 1 -- start start 2023-11-24T06:18:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31f676700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31815e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31f676700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3180ce430 0x7fd3181540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31f676700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31815e080 0x7fd3180cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31f676700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd318154620 con 0x7fd31815e080 2023-11-24T06:18:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31f676700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180b5b40 con 0x7fd3180ce430 2023-11-24T06:18:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31f676700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180b5e20 con 0x7fd3180b4310 2023-11-24T06:18:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31cc11700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3180ce430 0x7fd3181540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42894/0 (socket says 172.21.15.123:42894) 2023-11-24T06:18:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 -- 172.21.15.123:0/881176377 learned_addr learned my addr 172.21.15.123:0/881176377 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31dc13700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31815e080 0x7fd3180cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 -- 172.21.15.123:0/881176377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3180ce430 msgr2=0x7fd3181540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3180ce430 0x7fd3181540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 -- 172.21.15.123:0/881176377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31815e080 msgr2=0x7fd3180cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31815e080 0x7fd3180cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 -- 172.21.15.123:0/881176377 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3080094a0 con 0x7fd3180b4310 2023-11-24T06:18:25.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31cc11700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3180ce430 0x7fd3181540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.057+0000 7fd31d412700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31815e460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd31400c550 tx=0x7fd31400c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.058+0000 7fd30dffb700 1 -- 172.21.15.123:0/881176377 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314004c80 con 0x7fd3180b4310 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.058+0000 7fd30dffb700 1 -- 172.21.15.123:0/881176377 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd314004e00 con 0x7fd3180b4310 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.058+0000 7fd30dffb700 1 -- 172.21.15.123:0/881176377 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314007310 con 0x7fd3180b4310 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.058+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3180b6100 con 0x7fd3180b4310 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.058+0000 7fd30dffb700 1 -- 172.21.15.123:0/881176377 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd3140133c0 con 0x7fd3180b4310 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.058+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3180b66c0 con 0x7fd3180b4310 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.058+0000 7fd30dffb700 1 -- 172.21.15.123:0/881176377 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fd314013c00 con 0x7fd3180b4310 2023-11-24T06:18:25.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.059+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3180b6a30 con 0x7fd3180b4310 2023-11-24T06:18:25.066 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.061+0000 7fd30dffb700 1 -- 172.21.15.123:0/881176377 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fd3140123b0 con 0x7fd3180b4310 2023-11-24T06:18:25.072 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:25.070+0000 7f9c13b1a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-24T06:18:25.150 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.147+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "97e78a03-85d2-414c-8968-f999dd8e1126", "id": 1} v 0) v1 -- 0x7fd3180b6850 con 0x7fd3180b4310 2023-11-24T06:18:25.157 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.155+0000 7fd30dffb700 1 -- 172.21.15.123:0/881176377 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "97e78a03-85d2-414c-8968-f999dd8e1126", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd314012550 con 0x7fd3180b4310 2023-11-24T06:18:25.158 INFO:teuthology.orchestra.run.smithi123.stdout:1 2023-11-24T06:18:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 msgr2=0x7fd31815e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31815e460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd31400c550 tx=0x7fd31400c860 comp rx=0 tx=0).stop 2023-11-24T06:18:25.160 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 shutdown_connections 2023-11-24T06:18:25.160 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31815e080 0x7fd3180cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.161 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3180b4310 0x7fd31815e460 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.161 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 --2- 172.21.15.123:0/881176377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3180ce430 0x7fd3181540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.162 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 >> 172.21.15.123:0/881176377 conn(0x7fd3180b3780 msgr2=0x7fd31806f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:25.163 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 shutdown_connections 2023-11-24T06:18:25.163 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.156+0000 7fd31f676700 1 -- 172.21.15.123:0/881176377 wait complete. 2023-11-24T06:18:25.168 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:25.165+0000 7fab736cf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-24T06:18:25.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new b7abce2a-3e50-4e84-b63c-8758106b1da9 2 2023-11-24T06:18:25.193 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:25.191+0000 7f9c13b1a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-24T06:18:25.246 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:25.243+0000 7fab736cf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-24T06:18:25.283 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:25.280+0000 7f9c13b1a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-24T06:18:25.292 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.289+0000 7fb872aad700 1 Processor -- start 2023-11-24T06:18:25.292 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.289+0000 7fb872aad700 1 -- start start 2023-11-24T06:18:25.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb872aad700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb86c15bec0 0x7fb86c06fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb872aad700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c0d21d0 0x7fb86c0af520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb872aad700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0cc4a0 con 0x7fb86c0702e0 2023-11-24T06:18:25.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb872aad700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c070770 con 0x7fb86c15bec0 2023-11-24T06:18:25.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb872aad700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0d03e0 con 0x7fb86c0d21d0 2023-11-24T06:18:25.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb87104a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c0d21d0 0x7fb86c0af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb87104a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c0d21d0 0x7fb86c0af520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42904/0 (socket says 172.21.15.123:42904) 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb87104a700 1 -- 172.21.15.123:0/3615306890 learned_addr learned my addr 172.21.15.123:0/3615306890 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb870849700 1 --2- 172.21.15.123:0/3615306890 >> v2:172.21.15.89:3300/0 conn(0x7fb86c15bec0 0x7fb86c06fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb87104a700 1 -- 172.21.15.123:0/3615306890 >> v2:172.21.15.89:3300/0 conn(0x7fb86c15bec0 msgr2=0x7fb86c06fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb87104a700 1 --2- 172.21.15.123:0/3615306890 >> v2:172.21.15.89:3300/0 conn(0x7fb86c15bec0 0x7fb86c06fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb87104a700 1 -- 172.21.15.123:0/3615306890 >> v1:172.21.15.89:6789/0 conn(0x7fb86c0702e0 legacy=0x7fb86c0d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb87104a700 1 -- 172.21.15.123:0/3615306890 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c15c2a0 con 0x7fb86c0d21d0 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.290+0000 7fb87104a700 1 --2- 172.21.15.123:0/3615306890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c0d21d0 0x7fb86c0af520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb864009e50 tx=0x7fb86400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4267eb9a5fd4a48 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb86b7fe700 1 -- 172.21.15.123:0/3615306890 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86400e070 con 0x7fb86c0d21d0 2023-11-24T06:18:25.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb86b7fe700 1 -- 172.21.15.123:0/3615306890 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86400b910 con 0x7fb86c0d21d0 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb86b7fe700 1 -- 172.21.15.123:0/3615306890 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86400bcc0 con 0x7fb86c0d21d0 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb872aad700 1 -- 172.21.15.123:0/3615306890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c0d21d0 msgr2=0x7fb86c0af520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb872aad700 1 --2- 172.21.15.123:0/3615306890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c0d21d0 0x7fb86c0af520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb864009e50 tx=0x7fb86400b330 comp rx=0 tx=0).stop 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb872aad700 1 -- 172.21.15.123:0/3615306890 shutdown_connections 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb872aad700 1 --2- 172.21.15.123:0/3615306890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c0d21d0 0x7fb86c0af520 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb872aad700 1 --2- 172.21.15.123:0/3615306890 >> v2:172.21.15.89:3300/0 conn(0x7fb86c15bec0 0x7fb86c06fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb872aad700 1 -- 172.21.15.123:0/3615306890 >> 172.21.15.123:0/3615306890 conn(0x7fb86c0730a0 msgr2=0x7fb86c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb872aad700 1 -- 172.21.15.123:0/3615306890 shutdown_connections 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.291+0000 7fb872aad700 1 -- 172.21.15.123:0/3615306890 wait complete. 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.292+0000 7fb872aad700 1 Processor -- start 2023-11-24T06:18:25.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.292+0000 7fb872aad700 1 -- start start 2023-11-24T06:18:25.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb872aad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb86c0702e0 0x7fb86c157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb872aad700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb86c0d21d0 0x7fb86c1576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb872aad700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c15bec0 0x7fb86c0c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb872aad700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0c6520 con 0x7fb86c0d21d0 2023-11-24T06:18:25.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb872aad700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c15c630 con 0x7fb86c0702e0 2023-11-24T06:18:25.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb872aad700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c15c910 con 0x7fb86c15bec0 2023-11-24T06:18:25.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb870849700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb86c0702e0 0x7fb86c157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb87104a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c15bec0 0x7fb86c0c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb870849700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb86c0702e0 0x7fb86c157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58358/0 (socket says 172.21.15.123:58358) 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb870849700 1 -- 172.21.15.123:0/1413485246 learned_addr learned my addr 172.21.15.123:0/1413485246 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.293+0000 7fb86bfff700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb86c0d21d0 0x7fb86c1576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb87104a700 1 -- 172.21.15.123:0/1413485246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb86c0702e0 msgr2=0x7fb86c157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb87104a700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb86c0702e0 0x7fb86c157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb87104a700 1 -- 172.21.15.123:0/1413485246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb86c0d21d0 msgr2=0x7fb86c1576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb87104a700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb86c0d21d0 0x7fb86c1576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb87104a700 1 -- 172.21.15.123:0/1413485246 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb864009b30 con 0x7fb86c15bec0 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb870849700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb86c0702e0 0x7fb86c157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb87104a700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c15bec0 0x7fb86c0c6ea0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb864015000 tx=0x7fb864004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb8697fa700 1 -- 172.21.15.123:0/1413485246 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb864004780 con 0x7fb86c15bec0 2023-11-24T06:18:25.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb8697fa700 1 -- 172.21.15.123:0/1413485246 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8640092e0 con 0x7fb86c15bec0 2023-11-24T06:18:25.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c15cbf0 con 0x7fb86c15bec0 2023-11-24T06:18:25.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb8697fa700 1 -- 172.21.15.123:0/1413485246 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86401c680 con 0x7fb86c15bec0 2023-11-24T06:18:25.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c15d1b0 con 0x7fb86c15bec0 2023-11-24T06:18:25.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb8697fa700 1 -- 172.21.15.123:0/1413485246 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb86401cae0 con 0x7fb86c15bec0 2023-11-24T06:18:25.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.294+0000 7fb8697fa700 1 -- 172.21.15.123:0/1413485246 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fb864025c80 con 0x7fb86c15bec0 2023-11-24T06:18:25.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.295+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c15d520 con 0x7fb86c15bec0 2023-11-24T06:18:25.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.297+0000 7fb8697fa700 1 -- 172.21.15.123:0/1413485246 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb864020070 con 0x7fb86c15bec0 2023-11-24T06:18:25.326 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:25.324+0000 7fab736cf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-24T06:18:25.373 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:25.370+0000 7f9c13b1a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-24T06:18:25.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.381+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b7abce2a-3e50-4e84-b63c-8758106b1da9", "id": 2} v 0) v1 -- 0x7fb86c15d340 con 0x7fb86c15bec0 2023-11-24T06:18:25.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.387+0000 7fb8697fa700 1 -- 172.21.15.123:0/1413485246 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7abce2a-3e50-4e84-b63c-8758106b1da9", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb864025450 con 0x7fb86c15bec0 2023-11-24T06:18:25.390 INFO:teuthology.orchestra.run.smithi123.stdout:2 2023-11-24T06:18:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c15bec0 msgr2=0x7fb86c0c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c15bec0 0x7fb86c0c6ea0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb864015000 tx=0x7fb864004ad0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 shutdown_connections 2023-11-24T06:18:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb86c15bec0 0x7fb86c0c6ea0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb86c0702e0 0x7fb86c157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 --2- 172.21.15.123:0/1413485246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb86c0d21d0 0x7fb86c1576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 >> 172.21.15.123:0/1413485246 conn(0x7fb86c0730a0 msgr2=0x7fb86c0b6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:25.393 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 shutdown_connections 2023-11-24T06:18:25.393 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.388+0000 7fb872aad700 1 -- 172.21.15.123:0/1413485246 wait complete. 2023-11-24T06:18:25.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 7286d2e5-69a1-4cae-9893-0d8f496e78bf 3 2023-11-24T06:18:25.567 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.564+0000 7f6b71695700 1 Processor -- start 2023-11-24T06:18:25.567 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.564+0000 7f6b71695700 1 -- start start 2023-11-24T06:18:25.569 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b71695700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b6c0746a0 0x7f6b6c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.570 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b71695700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.570 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b71695700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c0b3c60 con 0x7f6b6c15e160 2023-11-24T06:18:25.570 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b71695700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6c0b4500 con 0x7f6b6c0746a0 2023-11-24T06:18:25.570 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b71695700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0b58c0 con 0x7f6b6c0afc90 2023-11-24T06:18:25.570 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6affd700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f6b6c15e160 0x7f6b6c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:38590/0 (socket says 172.21.15.123:38590) 2023-11-24T06:18:25.570 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6affd700 1 -- 172.21.15.123:0/3126020302 learned_addr learned my addr 172.21.15.123:0/3126020302 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6b7fe700 1 --2- 172.21.15.123:0/3126020302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6a7fc700 1 --2- 172.21.15.123:0/3126020302 >> v2:172.21.15.89:3300/0 conn(0x7f6b6c0746a0 0x7f6b6c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b69ffb700 1 -- 172.21.15.123:0/3126020302 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165996421 0 0) 0x7f6b6c0b3c60 con 0x7f6b6c15e160 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b69ffb700 1 -- 172.21.15.123:0/3126020302 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b540035b0 con 0x7f6b6c15e160 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6b7fe700 1 -- 172.21.15.123:0/3126020302 >> v2:172.21.15.89:3300/0 conn(0x7f6b6c0746a0 msgr2=0x7f6b6c0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6b7fe700 1 --2- 172.21.15.123:0/3126020302 >> v2:172.21.15.89:3300/0 conn(0x7f6b6c0746a0 0x7f6b6c0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6b7fe700 1 -- 172.21.15.123:0/3126020302 >> v1:172.21.15.123:6789/0 conn(0x7f6b6c15e160 legacy=0x7f6b6c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6b7fe700 1 -- 172.21.15.123:0/3126020302 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c0b0120 con 0x7f6b6c0afc90 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b6b7fe700 1 --2- 172.21.15.123:0/3126020302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c06d930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009d90 tx=0x7f6b6400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b1ab7be81c20335 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b69ffb700 1 -- 172.21.15.123:0/3126020302 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6400e070 con 0x7f6b6c0afc90 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b69ffb700 1 -- 172.21.15.123:0/3126020302 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6400bb20 con 0x7f6b6c0afc90 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.565+0000 7f6b69ffb700 1 -- 172.21.15.123:0/3126020302 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b64012430 con 0x7f6b6c0afc90 2023-11-24T06:18:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.566+0000 7f6b71695700 1 -- 172.21.15.123:0/3126020302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 msgr2=0x7f6b6c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.566+0000 7f6b71695700 1 --2- 172.21.15.123:0/3126020302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c06d930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009d90 tx=0x7f6b6400b330 comp rx=0 tx=0).stop 2023-11-24T06:18:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.566+0000 7f6b71695700 1 -- 172.21.15.123:0/3126020302 shutdown_connections 2023-11-24T06:18:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.566+0000 7f6b71695700 1 --2- 172.21.15.123:0/3126020302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c06d930 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.566+0000 7f6b71695700 1 --2- 172.21.15.123:0/3126020302 >> v2:172.21.15.89:3300/0 conn(0x7f6b6c0746a0 0x7f6b6c0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.566+0000 7f6b71695700 1 -- 172.21.15.123:0/3126020302 >> 172.21.15.123:0/3126020302 conn(0x7f6b6c0d0bf0 msgr2=0x7f6b6c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.566+0000 7f6b71695700 1 -- 172.21.15.123:0/3126020302 shutdown_connections 2023-11-24T06:18:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.566+0000 7f6b71695700 1 -- 172.21.15.123:0/3126020302 wait complete. 2023-11-24T06:18:25.575 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.567+0000 7f6b71695700 1 Processor -- start 2023-11-24T06:18:25.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.567+0000 7f6b71695700 1 -- start start 2023-11-24T06:18:25.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b71695700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b6c0746a0 0x7f6b6c158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b71695700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c0add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b71695700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b6c15e160 0x7f6b6c0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b71695700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0c6aa0 con 0x7f6b6c15e160 2023-11-24T06:18:25.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b71695700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0c6d80 con 0x7f6b6c0746a0 2023-11-24T06:18:25.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b71695700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0c7060 con 0x7f6b6c0afc90 2023-11-24T06:18:25.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6a7fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c0add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b6c0746a0 0x7f6b6c158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6a7fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c0add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42920/0 (socket says 172.21.15.123:42920) 2023-11-24T06:18:25.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b6c0746a0 0x7f6b6c158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58370/0 (socket says 172.21.15.123:58370) 2023-11-24T06:18:25.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6a7fc700 1 -- 172.21.15.123:0/2607279460 learned_addr learned my addr 172.21.15.123:0/2607279460 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:25.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6b7fe700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b6c15e160 0x7f6b6c0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6affd700 1 -- 172.21.15.123:0/2607279460 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 msgr2=0x7f6b6c0add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6affd700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c0add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6affd700 1 -- 172.21.15.123:0/2607279460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b6c15e160 msgr2=0x7f6b6c0ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6affd700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b6c15e160 0x7f6b6c0ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.568+0000 7f6b6affd700 1 -- 172.21.15.123:0/2607279460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b64009a70 con 0x7f6b6c0746a0 2023-11-24T06:18:25.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b6a7fc700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c0add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:25.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b6affd700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b6c0746a0 0x7f6b6c158880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00c430 tx=0x7f6b5c00c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:25.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b537fe700 1 -- 172.21.15.123:0/2607279460 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c005d70 con 0x7f6b6c0746a0 2023-11-24T06:18:25.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b537fe700 1 -- 172.21.15.123:0/2607279460 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c0070d0 con 0x7f6b6c0746a0 2023-11-24T06:18:25.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c0c7340 con 0x7f6b6c0746a0 2023-11-24T06:18:25.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b537fe700 1 -- 172.21.15.123:0/2607279460 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c007460 con 0x7f6b6c0746a0 2023-11-24T06:18:25.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c158dc0 con 0x7f6b6c0746a0 2023-11-24T06:18:25.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b537fe700 1 -- 172.21.15.123:0/2607279460 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6b5c01c440 con 0x7f6b6c0746a0 2023-11-24T06:18:25.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.569+0000 7f6b537fe700 1 -- 172.21.15.123:0/2607279460 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f6b5c012890 con 0x7f6b6c0746a0 2023-11-24T06:18:25.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.570+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6c15f770 con 0x7f6b6c0746a0 2023-11-24T06:18:25.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.572+0000 7f6b537fe700 1 -- 172.21.15.123:0/2607279460 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f6b6c15f770 con 0x7f6b6c0746a0 2023-11-24T06:18:25.658 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.655+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7286d2e5-69a1-4cae-9893-0d8f496e78bf", "id": 3} v 0) v1 -- 0x7f6b6c15faf0 con 0x7f6b6c0746a0 2023-11-24T06:18:25.661 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.659+0000 7f6b537fe700 1 -- 172.21.15.123:0/2607279460 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7286d2e5-69a1-4cae-9893-0d8f496e78bf", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6b5c017070 con 0x7f6b6c0746a0 2023-11-24T06:18:25.661 INFO:teuthology.orchestra.run.smithi123.stdout:3 2023-11-24T06:18:25.662 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.659+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b6c0746a0 msgr2=0x7f6b6c158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.663 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.659+0000 7f6b71695700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b6c0746a0 0x7f6b6c158880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00c430 tx=0x7f6b5c00c740 comp rx=0 tx=0).stop 2023-11-24T06:18:25.663 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.659+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 shutdown_connections 2023-11-24T06:18:25.663 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.659+0000 7f6b71695700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b6c15e160 0x7f6b6c0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.663 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.659+0000 7f6b71695700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b6c0746a0 0x7f6b6c158880 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.663 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.659+0000 7f6b71695700 1 --2- 172.21.15.123:0/2607279460 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b6c0afc90 0x7f6b6c0add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.663 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.660+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 >> 172.21.15.123:0/2607279460 conn(0x7f6b6c0d0bf0 msgr2=0x7f6b6c073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:25.663 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.660+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 shutdown_connections 2023-11-24T06:18:25.663 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.660+0000 7f6b71695700 1 -- 172.21.15.123:0/2607279460 wait complete. 2023-11-24T06:18:25.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 42ec5b7e-0658-47eb-8f04-d6a205b93ab2 4 2023-11-24T06:18:25.750 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:25.747+0000 7fab736cf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-24T06:18:25.795 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.792+0000 7f5ea1217700 1 Processor -- start 2023-11-24T06:18:25.796 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5ea1217700 1 -- start start 2023-11-24T06:18:25.798 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5ea1217700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5e9c15e160 0x7f5e9c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.798 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5ea1217700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5e9c0afc90 0x7f5e9c073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.798 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5ea1217700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c0b3c60 con 0x7f5e9c0744d0 2023-11-24T06:18:25.798 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5ea1217700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9c0b4500 con 0x7f5e9c0afc90 2023-11-24T06:18:25.798 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5ea1217700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9c074940 con 0x7f5e9c15e160 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5e9ad9d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5e9c15e160 0x7f5e9c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5e9ad9d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5e9c15e160 0x7f5e9c0b5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58376/0 (socket says 172.21.15.123:58376) 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.793+0000 7f5e9ad9d700 1 -- 172.21.15.123:0/1768917762 learned_addr learned my addr 172.21.15.123:0/1768917762 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e9a59c700 1 --2- 172.21.15.123:0/1768917762 >> v2:172.21.15.89:3300/0 conn(0x7f5e9c0afc90 0x7f5e9c073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e9ad9d700 1 -- 172.21.15.123:0/1768917762 >> v2:172.21.15.89:3300/0 conn(0x7f5e9c0afc90 msgr2=0x7f5e9c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e9ad9d700 1 --2- 172.21.15.123:0/1768917762 >> v2:172.21.15.89:3300/0 conn(0x7f5e9c0afc90 0x7f5e9c073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e9ad9d700 1 -- 172.21.15.123:0/1768917762 >> v1:172.21.15.123:6789/0 conn(0x7f5e9c0744d0 legacy=0x7f5e9c06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e9ad9d700 1 -- 172.21.15.123:0/1768917762 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c0b58c0 con 0x7f5e9c15e160 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e9a59c700 1 --2- 172.21.15.123:0/1768917762 >> v2:172.21.15.89:3300/0 conn(0x7f5e9c0afc90 0x7f5e9c073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e9ad9d700 1 --2- 172.21.15.123:0/1768917762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c15e160 0x7f5e9c0b5da0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c09efa0 tx=0x7f5e8c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=515c9193e3422ad6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e99d9b700 1 -- 172.21.15.123:0/1768917762 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c00b870 con 0x7f5e9c15e160 2023-11-24T06:18:25.799 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e99d9b700 1 -- 172.21.15.123:0/1768917762 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c00b9d0 con 0x7f5e9c15e160 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.794+0000 7f5e99d9b700 1 -- 172.21.15.123:0/1768917762 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c00bd60 con 0x7f5e9c15e160 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.795+0000 7f5ea1217700 1 -- 172.21.15.123:0/1768917762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c15e160 msgr2=0x7f5e9c0b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.795+0000 7f5ea1217700 1 --2- 172.21.15.123:0/1768917762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c15e160 0x7f5e9c0b5da0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c09efa0 tx=0x7f5e8c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.795+0000 7f5ea1217700 1 -- 172.21.15.123:0/1768917762 shutdown_connections 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.795+0000 7f5ea1217700 1 --2- 172.21.15.123:0/1768917762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c15e160 0x7f5e9c0b5da0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.795+0000 7f5ea1217700 1 --2- 172.21.15.123:0/1768917762 >> v2:172.21.15.89:3300/0 conn(0x7f5e9c0afc90 0x7f5e9c073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.795+0000 7f5ea1217700 1 -- 172.21.15.123:0/1768917762 >> 172.21.15.123:0/1768917762 conn(0x7f5e9c0d0bf0 msgr2=0x7f5e9c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.795+0000 7f5ea1217700 1 -- 172.21.15.123:0/1768917762 shutdown_connections 2023-11-24T06:18:25.800 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.795+0000 7f5ea1217700 1 -- 172.21.15.123:0/1768917762 wait complete. 2023-11-24T06:18:25.803 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.796+0000 7f5ea1217700 1 Processor -- start 2023-11-24T06:18:25.803 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.796+0000 7f5ea1217700 1 -- start start 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5ea1217700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c0744d0 0x7f5e9c1f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5ea1217700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e9c0afc90 0x7f5e9c161e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5ea1217700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c15e160 0x7f5e9c162360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5ea1217700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c1f7f60 con 0x7f5e9c15e160 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5ea1217700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0c5bd0 con 0x7f5e9c0744d0 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5ea1217700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0c5eb0 con 0x7f5e9c0afc90 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5e9a59c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e9c0afc90 0x7f5e9c161e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5e9a59c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e9c0afc90 0x7f5e9c161e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42934/0 (socket says 172.21.15.123:42934) 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5e9a59c700 1 -- 172.21.15.123:0/484710985 learned_addr learned my addr 172.21.15.123:0/484710985 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:25.804 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5e9ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c0744d0 0x7f5e9c1f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.805 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5e9b59e700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c15e160 0x7f5e9c162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:25.805 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5e9a59c700 1 -- 172.21.15.123:0/484710985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c0744d0 msgr2=0x7f5e9c1f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.797+0000 7f5e9a59c700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c0744d0 0x7f5e9c1f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e9a59c700 1 -- 172.21.15.123:0/484710985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c15e160 msgr2=0x7f5e9c162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e9a59c700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c15e160 0x7f5e9c162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e9a59c700 1 -- 172.21.15.123:0/484710985 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c009550 con 0x7f5e9c0afc90 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e9ad9d700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c0744d0 0x7f5e9c1f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e9a59c700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e9c0afc90 0x7f5e9c161e20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000e810 tx=0x7f5e9000eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e837fe700 1 -- 172.21.15.123:0/484710985 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90013610 con 0x7f5e9c0afc90 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e837fe700 1 -- 172.21.15.123:0/484710985 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e90013770 con 0x7f5e9c0afc90 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c0c6190 con 0x7f5e9c0afc90 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e837fe700 1 -- 172.21.15.123:0/484710985 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90013b20 con 0x7f5e9c0afc90 2023-11-24T06:18:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c1f37e0 con 0x7f5e9c0afc90 2023-11-24T06:18:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e837fe700 1 -- 172.21.15.123:0/484710985 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5e9001c450 con 0x7f5e9c0afc90 2023-11-24T06:18:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.798+0000 7f5e837fe700 1 -- 172.21.15.123:0/484710985 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f5e90012970 con 0x7f5e9c0afc90 2023-11-24T06:18:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.799+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c068fb0 con 0x7f5e9c0afc90 2023-11-24T06:18:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.801+0000 7f5e837fe700 1 -- 172.21.15.123:0/484710985 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f5e90021060 con 0x7f5e9c0afc90 2023-11-24T06:18:25.837 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:25.834+0000 7f9c13b1a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-24T06:18:25.903 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.900+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "42ec5b7e-0658-47eb-8f04-d6a205b93ab2", "id": 4} v 0) v1 -- 0x7f5e9c0c66b0 con 0x7f5e9c0afc90 2023-11-24T06:18:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.906+0000 7f5e837fe700 1 -- 172.21.15.123:0/484710985 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42ec5b7e-0658-47eb-8f04-d6a205b93ab2", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5e9c0c66b0 con 0x7f5e9c0afc90 2023-11-24T06:18:25.908 INFO:teuthology.orchestra.run.smithi123.stdout:4 2023-11-24T06:18:25.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e9c0afc90 msgr2=0x7f5e9c161e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:25.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e9c0afc90 0x7f5e9c161e20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000e810 tx=0x7f5e9000eb20 comp rx=0 tx=0).stop 2023-11-24T06:18:25.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 shutdown_connections 2023-11-24T06:18:25.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c15e160 0x7f5e9c162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9c0744d0 0x7f5e9c1f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 --2- 172.21.15.123:0/484710985 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e9c0afc90 0x7f5e9c161e20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:25.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 >> 172.21.15.123:0/484710985 conn(0x7f5e9c0d0bf0 msgr2=0x7f5e9c15d730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:25.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 shutdown_connections 2023-11-24T06:18:25.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:25.907+0000 7f5ea1217700 1 -- 172.21.15.123:0/484710985 wait complete. 2023-11-24T06:18:25.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0 5 2023-11-24T06:18:26.040 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.037+0000 7f4a6e850700 1 Processor -- start 2023-11-24T06:18:26.040 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.037+0000 7f4a6e850700 1 -- start start 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a6e850700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a6806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a6e850700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4a68162800 0x7f4a6806c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a6e850700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6815f280 con 0x7f4a6815e540 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a6e850700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6815f090 con 0x7f4a68162800 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a6e850700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6815f560 con 0x7f4a6815ba30 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a67fff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a6806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a67fff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a6806c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42936/0 (socket says 172.21.15.123:42936) 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a67fff700 1 -- 172.21.15.123:0/3485552880 learned_addr learned my addr 172.21.15.123:0/3485552880 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a6cded700 1 --1- 172.21.15.123:0/3485552880 >> v1:172.21.15.123:6789/0 conn(0x7f4a6815e540 0x7f4a6815e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:38616/0 (socket says 172.21.15.123:38616) 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a677fe700 1 --2- 172.21.15.123:0/3485552880 >> v2:172.21.15.89:3300/0 conn(0x7f4a68162800 0x7f4a6806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a66ffd700 1 -- 172.21.15.123:0/3485552880 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031375244 0 0) 0x7f4a6815f280 con 0x7f4a6815e540 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a67fff700 1 -- 172.21.15.123:0/3485552880 >> v2:172.21.15.89:3300/0 conn(0x7f4a68162800 msgr2=0x7f4a6806c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a67fff700 1 --2- 172.21.15.123:0/3485552880 >> v2:172.21.15.89:3300/0 conn(0x7f4a68162800 0x7f4a6806c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a67fff700 1 -- 172.21.15.123:0/3485552880 >> v1:172.21.15.123:6789/0 conn(0x7f4a6815e540 legacy=0x7f4a6815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a67fff700 1 -- 172.21.15.123:0/3485552880 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68161d20 con 0x7f4a6815ba30 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.038+0000 7f4a677fe700 1 --2- 172.21.15.123:0/3485552880 >> v2:172.21.15.89:3300/0 conn(0x7f4a68162800 0x7f4a6806c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a67fff700 1 --2- 172.21.15.123:0/3485552880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a6806c000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a580098f0 tx=0x7f4a58009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e39036bf13b74bc5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a66ffd700 1 -- 172.21.15.123:0/3485552880 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5800b8a0 con 0x7f4a6815ba30 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a66ffd700 1 -- 172.21.15.123:0/3485552880 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5800ba00 con 0x7f4a6815ba30 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a66ffd700 1 -- 172.21.15.123:0/3485552880 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5800bd90 con 0x7f4a6815ba30 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a6e850700 1 -- 172.21.15.123:0/3485552880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 msgr2=0x7f4a6806c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a6e850700 1 --2- 172.21.15.123:0/3485552880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a6806c000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a580098f0 tx=0x7f4a58009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:26.044 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a6e850700 1 -- 172.21.15.123:0/3485552880 shutdown_connections 2023-11-24T06:18:26.045 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a6e850700 1 --2- 172.21.15.123:0/3485552880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a6806c000 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.045 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a6e850700 1 --2- 172.21.15.123:0/3485552880 >> v2:172.21.15.89:3300/0 conn(0x7f4a68162800 0x7f4a6806c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.045 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a6e850700 1 -- 172.21.15.123:0/3485552880 >> 172.21.15.123:0/3485552880 conn(0x7f4a68071a70 msgr2=0x7f4a68071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:26.045 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a6e850700 1 -- 172.21.15.123:0/3485552880 shutdown_connections 2023-11-24T06:18:26.045 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.039+0000 7f4a6e850700 1 -- 172.21.15.123:0/3485552880 wait complete. 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.041+0000 7f4a6e850700 1 Processor -- start 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.041+0000 7f4a6e850700 1 -- start start 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.041+0000 7f4a6e850700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a681f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.041+0000 7f4a6e850700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6815e540 0x7f4a680cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.041+0000 7f4a6e850700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a68162800 0x7f4a680b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.041+0000 7f4a6e850700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a680b3a80 con 0x7f4a6815e540 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.041+0000 7f4a6e850700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6813b9b0 con 0x7f4a68162800 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.041+0000 7f4a6e850700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a680cf900 con 0x7f4a6815ba30 2023-11-24T06:18:26.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a67fff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a681f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a67fff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a681f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42942/0 (socket says 172.21.15.123:42942) 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a67fff700 1 -- 172.21.15.123:0/1306888637 learned_addr learned my addr 172.21.15.123:0/1306888637 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6cded700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a68162800 0x7f4a680b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a677fe700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6815e540 0x7f4a680cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6cded700 1 -- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 msgr2=0x7f4a681f2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6cded700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a681f2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6cded700 1 -- 172.21.15.123:0/1306888637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6815e540 msgr2=0x7f4a680cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6cded700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6815e540 0x7f4a680cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6cded700 1 -- 172.21.15.123:0/1306888637 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a58009550 con 0x7f4a68162800 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a67fff700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a681f2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:26.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6cded700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a68162800 0x7f4a680b5db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000cbe0 tx=0x7f4a6000cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a64ff9700 1 -- 172.21.15.123:0/1306888637 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6000aa40 con 0x7f4a68162800 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a64ff9700 1 -- 172.21.15.123:0/1306888637 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6000abc0 con 0x7f4a68162800 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6815bf90 con 0x7f4a68162800 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a64ff9700 1 -- 172.21.15.123:0/1306888637 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600070d0 con 0x7f4a68162800 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.042+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a680cfec0 con 0x7f4a68162800 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.043+0000 7f4a64ff9700 1 -- 172.21.15.123:0/1306888637 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4a60007530 con 0x7f4a68162800 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.043+0000 7f4a64ff9700 1 -- 172.21.15.123:0/1306888637 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f4a60012980 con 0x7f4a68162800 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.043+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a68068fb0 con 0x7f4a68162800 2023-11-24T06:18:26.050 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.045+0000 7f4a64ff9700 1 -- 172.21.15.123:0/1306888637 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f4a60018070 con 0x7f4a68162800 2023-11-24T06:18:26.132 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.130+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0", "id": 5} v 0) v1 -- 0x7f4a681f33a0 con 0x7f4a68162800 2023-11-24T06:18:26.138 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.135+0000 7f4a64ff9700 1 -- 172.21.15.123:0/1306888637 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4a60007970 con 0x7f4a68162800 2023-11-24T06:18:26.138 INFO:teuthology.orchestra.run.smithi123.stdout:5 2023-11-24T06:18:26.139 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.136+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a68162800 msgr2=0x7f4a680b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.139 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.136+0000 7f4a6e850700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a68162800 0x7f4a680b5db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000cbe0 tx=0x7f4a6000cef0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.139 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.136+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 shutdown_connections 2023-11-24T06:18:26.140 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.136+0000 7f4a6e850700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a68162800 0x7f4a680b5db0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.140 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.136+0000 7f4a6e850700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a6815ba30 0x7f4a681f2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.140 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.136+0000 7f4a6e850700 1 --2- 172.21.15.123:0/1306888637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6815e540 0x7f4a680cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.140 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.136+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 >> 172.21.15.123:0/1306888637 conn(0x7f4a68071a70 msgr2=0x7f4a68071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:26.140 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.137+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 shutdown_connections 2023-11-24T06:18:26.140 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.137+0000 7f4a6e850700 1 -- 172.21.15.123:0/1306888637 wait complete. 2023-11-24T06:18:26.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 0317ea02-1e74-45a8-8734-eaa2f69e85ae 6 2023-11-24T06:18:26.299 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.296+0000 7ff34dde7700 1 Processor -- start 2023-11-24T06:18:26.299 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.296+0000 7ff34dde7700 1 -- start start 2023-11-24T06:18:26.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff34dde7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff3481456f0 0x7ff3480b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff34dde7700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff348145ad0 con 0x7ff34815ce40 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff34dde7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34815fd10 con 0x7ff3480746a0 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff34dde7700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34815fff0 con 0x7ff3481456f0 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff347fff700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff3480746a0 0x7ff34806b2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:38630/0 (socket says 172.21.15.123:38630) 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff347fff700 1 -- 172.21.15.123:0/510823958 learned_addr learned my addr 172.21.15.123:0/510823958 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff3477fe700 1 --2- 172.21.15.123:0/510823958 >> v2:172.21.15.89:3300/0 conn(0x7ff3481456f0 0x7ff3480b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff3467fc700 1 -- 172.21.15.123:0/510823958 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225778014 0 0) 0x7ff34815fd10 con 0x7ff3480746a0 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff3467fc700 1 -- 172.21.15.123:0/510823958 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3300035b0 con 0x7ff3480746a0 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff3477fe700 1 -- 172.21.15.123:0/510823958 >> v1:172.21.15.123:6789/0 conn(0x7ff3480746a0 legacy=0x7ff34806b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff3477fe700 1 -- 172.21.15.123:0/510823958 >> v1:172.21.15.89:6789/0 conn(0x7ff34815ce40 legacy=0x7ff348073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.303 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff3467fc700 1 -- 172.21.15.123:0/510823958 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377335707 0 0) 0x7ff348145ad0 con 0x7ff34815ce40 2023-11-24T06:18:26.303 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.298+0000 7ff3477fe700 1 -- 172.21.15.123:0/510823958 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3481602d0 con 0x7ff3481456f0 2023-11-24T06:18:26.303 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.299+0000 7ff3477fe700 1 --2- 172.21.15.123:0/510823958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3481456f0 0x7ff3480b3bf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff340009dc0 tx=0x7ff34000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604d7aec7c4df73e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:26.303 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.299+0000 7ff3467fc700 1 -- 172.21.15.123:0/510823958 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34000e070 con 0x7ff3481456f0 2023-11-24T06:18:26.303 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.299+0000 7ff3467fc700 1 -- 172.21.15.123:0/510823958 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34000bae0 con 0x7ff3481456f0 2023-11-24T06:18:26.303 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.299+0000 7ff3467fc700 1 -- 172.21.15.123:0/510823958 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34000be90 con 0x7ff3481456f0 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.300+0000 7ff34dde7700 1 -- 172.21.15.123:0/510823958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3481456f0 msgr2=0x7ff3480b3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.300+0000 7ff34dde7700 1 --2- 172.21.15.123:0/510823958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3481456f0 0x7ff3480b3bf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff340009dc0 tx=0x7ff34000b330 comp rx=0 tx=0).stop 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.300+0000 7ff34dde7700 1 -- 172.21.15.123:0/510823958 shutdown_connections 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.300+0000 7ff34dde7700 1 --2- 172.21.15.123:0/510823958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3481456f0 0x7ff3480b3bf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.300+0000 7ff34dde7700 1 -- 172.21.15.123:0/510823958 >> 172.21.15.123:0/510823958 conn(0x7ff3480b28f0 msgr2=0x7ff3480b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.300+0000 7ff34dde7700 1 -- 172.21.15.123:0/510823958 shutdown_connections 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.300+0000 7ff34dde7700 1 -- 172.21.15.123:0/510823958 wait complete. 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.301+0000 7ff34dde7700 1 Processor -- start 2023-11-24T06:18:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.301+0000 7ff34dde7700 1 -- start start 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff34dde7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3480746a0 0x7ff34809e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff34dde7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3481456f0 0x7ff3480b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff34dde7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff34815ce40 0x7ff3480b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff34dde7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34813cda0 con 0x7ff34815ce40 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff34dde7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34814b240 con 0x7ff3480746a0 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff34dde7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34813cf10 con 0x7ff3481456f0 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff3477fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3480746a0 0x7ff34809e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff346ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3481456f0 0x7ff3480b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff3477fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3480746a0 0x7ff34809e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58406/0 (socket says 172.21.15.123:58406) 2023-11-24T06:18:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff3477fe700 1 -- 172.21.15.123:0/523500202 learned_addr learned my addr 172.21.15.123:0/523500202 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff347fff700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff34815ce40 0x7ff3480b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff346ffd700 1 -- 172.21.15.123:0/523500202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3480746a0 msgr2=0x7ff34809e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff346ffd700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3480746a0 0x7ff34809e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff346ffd700 1 -- 172.21.15.123:0/523500202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff34815ce40 msgr2=0x7ff3480b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff346ffd700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff34815ce40 0x7ff3480b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff346ffd700 1 -- 172.21.15.123:0/523500202 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff340009a70 con 0x7ff3481456f0 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff3477fe700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3480746a0 0x7ff34809e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff346ffd700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3481456f0 0x7ff3480b3f50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00c410 tx=0x7ff33c00c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff32ffff700 1 -- 172.21.15.123:0/523500202 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0052d0 con 0x7ff3481456f0 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff32ffff700 1 -- 172.21.15.123:0/523500202 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c005430 con 0x7ff3481456f0 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34814e090 con 0x7ff3481456f0 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.302+0000 7ff32ffff700 1 -- 172.21.15.123:0/523500202 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0038a0 con 0x7ff3481456f0 2023-11-24T06:18:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.303+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34814b770 con 0x7ff3481456f0 2023-11-24T06:18:26.309 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.303+0000 7ff32ffff700 1 -- 172.21.15.123:0/523500202 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff33c01c440 con 0x7ff3481456f0 2023-11-24T06:18:26.309 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.303+0000 7ff32ffff700 1 -- 172.21.15.123:0/523500202 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ff33c01cce0 con 0x7ff3481456f0 2023-11-24T06:18:26.309 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.304+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34809e970 con 0x7ff3481456f0 2023-11-24T06:18:26.309 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.306+0000 7ff32ffff700 1 -- 172.21.15.123:0/523500202 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff33c017070 con 0x7ff3481456f0 2023-11-24T06:18:26.379 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:26.376+0000 7fab736cf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-24T06:18:26.392 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.390+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0317ea02-1e74-45a8-8734-eaa2f69e85ae", "id": 6} v 0) v1 -- 0x7ff348068fb0 con 0x7ff3481456f0 2023-11-24T06:18:26.400 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.398+0000 7ff32ffff700 1 -- 172.21.15.123:0/523500202 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0317ea02-1e74-45a8-8734-eaa2f69e85ae", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff33c01c7c0 con 0x7ff3481456f0 2023-11-24T06:18:26.401 INFO:teuthology.orchestra.run.smithi123.stdout:6 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3481456f0 msgr2=0x7ff3480b3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3481456f0 0x7ff3480b3f50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00c410 tx=0x7ff33c00c720 comp rx=0 tx=0).stop 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 shutdown_connections 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff34815ce40 0x7ff3480b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3480746a0 0x7ff34809e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 --2- 172.21.15.123:0/523500202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3481456f0 0x7ff3480b3f50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 >> 172.21.15.123:0/523500202 conn(0x7ff3480b28f0 msgr2=0x7ff3480b5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 shutdown_connections 2023-11-24T06:18:26.402 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.399+0000 7ff34dde7700 1 -- 172.21.15.123:0/523500202 wait complete. 2023-11-24T06:18:26.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 1f14683e-7e01-4616-9c01-3dafbbda1f0e 7 2023-11-24T06:18:26.448 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:26.445+0000 7fab736cf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-24T06:18:26.486 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:26.484+0000 7f9c13b1a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-24T06:18:26.535 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.533+0000 7f929b4c4700 1 Processor -- start 2023-11-24T06:18:26.536 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.533+0000 7f929b4c4700 1 -- start start 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f929b4c4700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f929415e160 0x7f92940b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f929b4c4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92940afc90 0x7f9294073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f929b4c4700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f92940744d0 0x7f929406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f929b4c4700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92940b4500 con 0x7f929415e160 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f929b4c4700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9294074960 con 0x7f92940744d0 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f929b4c4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92940b3c60 con 0x7f92940afc90 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299a61700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f92940744d0 0x7f929406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9298a5f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92940afc90 0x7f9294073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9298a5f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92940afc90 0x7f9294073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:42962/0 (socket says 172.21.15.123:42962) 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299a61700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f92940744d0 0x7f929406d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58416/0 (socket says 172.21.15.123:58416) 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9298a5f700 1 -- 172.21.15.123:0/1489456981 learned_addr learned my addr 172.21.15.123:0/1489456981 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299260700 1 --2- 172.21.15.123:0/1489456981 >> v2:172.21.15.89:3300/0 conn(0x7f929415e160 0x7f92940b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299a61700 1 -- 172.21.15.123:0/1489456981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92940afc90 msgr2=0x7f9294073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299a61700 1 --2- 172.21.15.123:0/1489456981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92940afc90 0x7f9294073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299a61700 1 -- 172.21.15.123:0/1489456981 >> v2:172.21.15.89:3300/0 conn(0x7f929415e160 msgr2=0x7f92940b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299a61700 1 --2- 172.21.15.123:0/1489456981 >> v2:172.21.15.89:3300/0 conn(0x7f929415e160 0x7f92940b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299a61700 1 -- 172.21.15.123:0/1489456981 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92940b58c0 con 0x7f92940744d0 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9298a5f700 1 --2- 172.21.15.123:0/1489456981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92940afc90 0x7f9294073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f9299a61700 1 --2- 172.21.15.123:0/1489456981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940744d0 0x7f929406d930 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f92940d1cd0 tx=0x7f928c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc0f5244b389e50a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f928bfff700 1 -- 172.21.15.123:0/1489456981 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c00e070 con 0x7f92940744d0 2023-11-24T06:18:26.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f928bfff700 1 -- 172.21.15.123:0/1489456981 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c00bb20 con 0x7f92940744d0 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.534+0000 7f928bfff700 1 -- 172.21.15.123:0/1489456981 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c012430 con 0x7f92940744d0 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 -- 172.21.15.123:0/1489456981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940744d0 msgr2=0x7f929406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 --2- 172.21.15.123:0/1489456981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940744d0 0x7f929406d930 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f92940d1cd0 tx=0x7f928c00b330 comp rx=0 tx=0).stop 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 -- 172.21.15.123:0/1489456981 shutdown_connections 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 --2- 172.21.15.123:0/1489456981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940744d0 0x7f929406d930 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 --2- 172.21.15.123:0/1489456981 >> v2:172.21.15.89:3300/0 conn(0x7f929415e160 0x7f92940b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 --2- 172.21.15.123:0/1489456981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92940afc90 0x7f9294073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 -- 172.21.15.123:0/1489456981 >> 172.21.15.123:0/1489456981 conn(0x7f92940d0bf0 msgr2=0x7f92940d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 -- 172.21.15.123:0/1489456981 shutdown_connections 2023-11-24T06:18:26.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.535+0000 7f929b4c4700 1 -- 172.21.15.123:0/1489456981 wait complete. 2023-11-24T06:18:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.536+0000 7f929b4c4700 1 Processor -- start 2023-11-24T06:18:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.536+0000 7f929b4c4700 1 -- start start 2023-11-24T06:18:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f929b4c4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92940744d0 0x7f9294145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f929b4c4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940afc90 0x7f9294160a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f929b4c4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f929415e160 0x7f9294156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f929b4c4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92941568d0 con 0x7f92940744d0 2023-11-24T06:18:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f929b4c4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9294155cc0 con 0x7f92940afc90 2023-11-24T06:18:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f929b4c4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9294155fa0 con 0x7f929415e160 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f9298a5f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940afc90 0x7f9294160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f9298a5f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940afc90 0x7f9294160a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58420/0 (socket says 172.21.15.123:58420) 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f9298a5f700 1 -- 172.21.15.123:0/3629390502 learned_addr learned my addr 172.21.15.123:0/3629390502 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f9299a61700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f929415e160 0x7f9294156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.537+0000 7f9299260700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92940744d0 0x7f9294145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9298a5f700 1 -- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f929415e160 msgr2=0x7f9294156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9298a5f700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f929415e160 0x7f9294156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9298a5f700 1 -- 172.21.15.123:0/3629390502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92940744d0 msgr2=0x7f9294145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9298a5f700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92940744d0 0x7f9294145090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9298a5f700 1 -- 172.21.15.123:0/3629390502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9284009550 con 0x7f92940afc90 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9299a61700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f929415e160 0x7f9294156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9298a5f700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940afc90 0x7f9294160a10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f928400ebe0 tx=0x7f928400eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9289ffb700 1 -- 172.21.15.123:0/3629390502 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928400c850 con 0x7f92940afc90 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9289ffb700 1 -- 172.21.15.123:0/3629390502 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928400c9b0 con 0x7f92940afc90 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928c009a70 con 0x7f92940afc90 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9289ffb700 1 -- 172.21.15.123:0/3629390502 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928400cd40 con 0x7f92940afc90 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9294156560 con 0x7f92940afc90 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9289ffb700 1 -- 172.21.15.123:0/3629390502 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9284005d20 con 0x7f92940afc90 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.538+0000 7f9289ffb700 1 -- 172.21.15.123:0/3629390502 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f9284006840 con 0x7f92940afc90 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.539+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92940af840 con 0x7f92940afc90 2023-11-24T06:18:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.541+0000 7f9289ffb700 1 -- 172.21.15.123:0/3629390502 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9284021060 con 0x7f92940afc90 2023-11-24T06:18:26.559 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:26.557+0000 7f9c13b1a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-24T06:18:26.628 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.626+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1f14683e-7e01-4616-9c01-3dafbbda1f0e", "id": 7} v 0) v1 -- 0x7f9294068fb0 con 0x7f92940afc90 2023-11-24T06:18:26.634 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.631+0000 7f9289ffb700 1 -- 172.21.15.123:0/3629390502 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1f14683e-7e01-4616-9c01-3dafbbda1f0e", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9284017070 con 0x7f92940afc90 2023-11-24T06:18:26.634 INFO:teuthology.orchestra.run.smithi123.stdout:7 2023-11-24T06:18:26.635 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940afc90 msgr2=0x7f9294160a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.635 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940afc90 0x7f9294160a10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f928400ebe0 tx=0x7f928400eef0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.635 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 shutdown_connections 2023-11-24T06:18:26.635 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f929415e160 0x7f9294156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92940744d0 0x7f9294145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 --2- 172.21.15.123:0/3629390502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92940afc90 0x7f9294160a10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 >> 172.21.15.123:0/3629390502 conn(0x7f92940d0bf0 msgr2=0x7f929406deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 shutdown_connections 2023-11-24T06:18:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-24T06:18:26.632+0000 7f929b4c4700 1 -- 172.21.15.123:0/3629390502 wait complete. 2023-11-24T06:18:26.646 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-24T06:18:26.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-24T06:18:26.649 INFO:tasks.ceph.osd.0:Started 2023-11-24T06:18:26.649 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-24T06:18:26.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-24T06:18:26.651 INFO:tasks.ceph.osd.1:Started 2023-11-24T06:18:26.651 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-24T06:18:26.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-24T06:18:26.653 INFO:tasks.ceph.osd.2:Started 2023-11-24T06:18:26.654 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-24T06:18:26.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-24T06:18:26.655 INFO:tasks.ceph.osd.3:Started 2023-11-24T06:18:26.656 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-24T06:18:26.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-24T06:18:26.658 INFO:tasks.ceph.osd.4:Started 2023-11-24T06:18:26.658 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-24T06:18:26.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-24T06:18:26.660 INFO:tasks.ceph.osd.5:Started 2023-11-24T06:18:26.660 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-24T06:18:26.660 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-24T06:18:26.662 INFO:tasks.ceph.osd.6:Started 2023-11-24T06:18:26.662 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-24T06:18:26.662 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-24T06:18:26.664 INFO:tasks.ceph.osd.7:Started 2023-11-24T06:18:26.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.811+0000 7f6f711a4700 1 Processor -- start 2023-11-24T06:18:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.811+0000 7f6f711a4700 1 -- start start 2023-11-24T06:18:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f711a4700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f6c0b0ba0 0x7f6f6c0af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f711a4700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f6c0b0f80 0x7f6f6c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f711a4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c15ee90 0x7f6f6c0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f711a4700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6c0b2bd0 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f711a4700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6c0b1fc0 con 0x7f6f6c0b0ba0 2023-11-24T06:18:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f711a4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c0b22a0 con 0x7f6f6c15ee90 2023-11-24T06:18:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6a59c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f6c0b0f80 0x7f6f6c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6a59c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f6c0b0f80 0x7f6f6c0cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34402/0 (socket says 172.21.15.89:34402) 2023-11-24T06:18:26.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6a59c700 1 -- 172.21.15.89:0/1345874732 learned_addr learned my addr 172.21.15.89:0/1345874732 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:26.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6ad9d700 1 --2- 172.21.15.89:0/1345874732 >> v2:172.21.15.123:3300/0 conn(0x7f6f6c0b0ba0 0x7f6f6c0af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6b59e700 1 --2- 172.21.15.89:0/1345874732 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c15ee90 0x7f6f6c0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6a59c700 1 -- 172.21.15.89:0/1345874732 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c15ee90 msgr2=0x7f6f6c0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6a59c700 1 --2- 172.21.15.89:0/1345874732 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c15ee90 0x7f6f6c0ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6a59c700 1 -- 172.21.15.89:0/1345874732 >> v2:172.21.15.123:3300/0 conn(0x7f6f6c0b0ba0 msgr2=0x7f6f6c0af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6a59c700 1 --2- 172.21.15.89:0/1345874732 >> v2:172.21.15.123:3300/0 conn(0x7f6f6c0b0ba0 0x7f6f6c0af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.812+0000 7f6f6a59c700 1 -- 172.21.15.89:0/1345874732 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6c0b2580 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f6a59c700 1 --2- 172.21.15.89:0/1345874732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c0cc760 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c0d1340 tx=0x7f6f60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9fee4073289d61 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f69d9b700 1 -- 172.21.15.89:0/1345874732 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60011830 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f69d9b700 1 -- 172.21.15.89:0/1345874732 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f60011990 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f69d9b700 1 -- 172.21.15.89:0/1345874732 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60011d20 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 -- 172.21.15.89:0/1345874732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 msgr2=0x7f6f6c0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 --2- 172.21.15.89:0/1345874732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c0cc760 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c0d1340 tx=0x7f6f60009bd0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 -- 172.21.15.89:0/1345874732 shutdown_connections 2023-11-24T06:18:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 --2- 172.21.15.89:0/1345874732 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c15ee90 0x7f6f6c0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 --2- 172.21.15.89:0/1345874732 >> v2:172.21.15.123:3300/0 conn(0x7f6f6c0b0ba0 0x7f6f6c0af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 --2- 172.21.15.89:0/1345874732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c0cc760 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 -- 172.21.15.89:0/1345874732 >> 172.21.15.89:0/1345874732 conn(0x7f6f6c0b57b0 msgr2=0x7f6f6c0b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 -- 172.21.15.89:0/1345874732 shutdown_connections 2023-11-24T06:18:26.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.813+0000 7f6f711a4700 1 -- 172.21.15.89:0/1345874732 wait complete. 2023-11-24T06:18:26.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.815+0000 7f6f711a4700 1 Processor -- start 2023-11-24T06:18:26.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.815+0000 7f6f711a4700 1 -- start start 2023-11-24T06:18:26.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f711a4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c0b0ba0 0x7f6f6c149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f711a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c1e4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f711a4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f6c15ee90 0x7f6f6c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:26.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f711a4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1f5370 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f711a4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1f46f0 con 0x7f6f6c15ee90 2023-11-24T06:18:26.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f711a4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1f49d0 con 0x7f6f6c0b0ba0 2023-11-24T06:18:26.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c1e4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c1e4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34408/0 (socket says 172.21.15.89:34408) 2023-11-24T06:18:26.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 -- 172.21.15.89:0/2663304990 learned_addr learned my addr 172.21.15.89:0/2663304990 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:26.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6ad9d700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c0b0ba0 0x7f6f6c149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6b59e700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f6c15ee90 0x7f6f6c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:26.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 -- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c0b0ba0 msgr2=0x7f6f6c149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c0b0ba0 0x7f6f6c149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 -- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f6c15ee90 msgr2=0x7f6f6c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f6c15ee90 0x7f6f6c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 -- 172.21.15.89:0/2663304990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60009550 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6ad9d700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c0b0ba0 0x7f6f6c149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:26.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f6a59c700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c1e4700 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6f60000c00 tx=0x7f6f60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:26.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f537fe700 1 -- 172.21.15.89:0/2663304990 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60004240 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f537fe700 1 -- 172.21.15.89:0/2663304990 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6001f070 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6c1f4cb0 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.816+0000 7f6f537fe700 1 -- 172.21.15.89:0/2663304990 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f600036a0 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.817+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c149ba0 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.817+0000 7f6f537fe700 1 -- 172.21.15.89:0/2663304990 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6f60021030 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.817+0000 7f6f537fe700 1 -- 172.21.15.89:0/2663304990 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6f6001ada0 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.817+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6c0b1690 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.819+0000 7f6f537fe700 1 -- 172.21.15.89:0/2663304990 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f6f60015020 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.890+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f6c067f60 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.890+0000 7f6f537fe700 1 -- 172.21.15.89:0/2663304990 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6f60034090 con 0x7f6f6c0b0f80 2023-11-24T06:18:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:26.894 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":10,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:26.628120+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:26.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 msgr2=0x7f6f6c1e4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:26.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c1e4700 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6f60000c00 tx=0x7f6f60004750 comp rx=0 tx=0).stop 2023-11-24T06:18:26.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 shutdown_connections 2023-11-24T06:18:26.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f6c15ee90 0x7f6f6c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c0b0ba0 0x7f6f6c149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 --2- 172.21.15.89:0/2663304990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f6c0b0f80 0x7f6f6c1e4700 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:26.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 >> 172.21.15.89:0/2663304990 conn(0x7f6f6c0b57b0 msgr2=0x7f6f6c0d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:26.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 shutdown_connections 2023-11-24T06:18:26.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:26.891+0000 7f6f711a4700 1 -- 172.21.15.89:0/2663304990 wait complete. 2023-11-24T06:18:26.906 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-24T06:18:26.907 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-24T06:18:27.195 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:27.192+0000 7fab736cf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-24T06:18:27.264 INFO:tasks.ceph.osd.7.smithi123.stderr:2023-11-24T06:18:27.261+0000 7f63ef145640 -1 Falling back to public interface 2023-11-24T06:18:27.268 INFO:tasks.ceph.osd.5.smithi123.stderr:2023-11-24T06:18:27.266+0000 7f8e532e9640 -1 Falling back to public interface 2023-11-24T06:18:27.272 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:27.269+0000 7fab736cf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-24T06:18:27.274 INFO:tasks.ceph.osd.0.smithi089.stderr:2023-11-24T06:18:27.271+0000 7f38f25b3640 -1 Falling back to public interface 2023-11-24T06:18:27.279 INFO:tasks.ceph.osd.4.smithi123.stderr:2023-11-24T06:18:27.276+0000 7fc715699640 -1 Falling back to public interface 2023-11-24T06:18:27.279 INFO:tasks.ceph.osd.1.smithi089.stderr:2023-11-24T06:18:27.276+0000 7f6266296640 -1 Falling back to public interface 2023-11-24T06:18:27.279 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:27.277+0000 7f9c13b1a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-24T06:18:27.283 INFO:tasks.ceph.osd.2.smithi089.stderr:2023-11-24T06:18:27.280+0000 7f351510d640 -1 Falling back to public interface 2023-11-24T06:18:27.288 INFO:tasks.ceph.osd.3.smithi089.stderr:2023-11-24T06:18:27.285+0000 7f81cd9be640 -1 Falling back to public interface 2023-11-24T06:18:27.289 INFO:tasks.ceph.osd.6.smithi123.stderr:2023-11-24T06:18:27.286+0000 7f2cf2bb9640 -1 Falling back to public interface 2023-11-24T06:18:27.359 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:27.356+0000 7f9c13b1a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-24T06:18:27.418 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:27.415+0000 7fab736cf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-24T06:18:27.499 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:27.497+0000 7f9c13b1a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-24T06:18:27.655 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:27.652+0000 7fab736cf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-24T06:18:27.711 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:27.736 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:27.733+0000 7f9c13b1a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-24T06:18:27.827 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.824+0000 7f8aabb6a700 1 Processor -- start 2023-11-24T06:18:27.827 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.824+0000 7f8aabb6a700 1 -- start start 2023-11-24T06:18:27.830 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.825+0000 7f8aabb6a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8aa415bf60 0x7f8aa40cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:27.830 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.825+0000 7f8aabb6a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa40d0c40 con 0x7f8aa40d0740 2023-11-24T06:18:27.831 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.825+0000 7f8aabb6a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa406f220 con 0x7f8aa40d02c0 2023-11-24T06:18:27.831 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.825+0000 7f8aabb6a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa415c340 con 0x7f8aa415bf60 2023-11-24T06:18:27.831 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.825+0000 7f8aaa107700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8aa40d0740 0x7f8aa4070230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:37920/0 (socket says 172.21.15.89:37920) 2023-11-24T06:18:27.831 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.825+0000 7f8aaa107700 1 -- 172.21.15.89:0/3955981729 learned_addr learned my addr 172.21.15.89:0/3955981729 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:27.831 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa9906700 1 --2- 172.21.15.89:0/3955981729 >> v2:172.21.15.123:3300/0 conn(0x7f8aa415bf60 0x7f8aa40cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:27.831 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa8904700 1 -- 172.21.15.89:0/3955981729 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523753647 0 0) 0x7f8aa40d0c40 con 0x7f8aa40d0740 2023-11-24T06:18:27.831 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa8904700 1 -- 172.21.15.89:0/3955981729 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a900035b0 con 0x7f8aa40d0740 2023-11-24T06:18:27.831 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa9906700 1 -- 172.21.15.89:0/3955981729 >> v1:172.21.15.123:6789/0 conn(0x7f8aa40d02c0 legacy=0x7f8aa40d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:27.832 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa8904700 1 -- 172.21.15.89:0/3955981729 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559977744 0 0) 0x7f8a900035b0 con 0x7f8aa40d0740 2023-11-24T06:18:27.832 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa9906700 1 -- 172.21.15.89:0/3955981729 >> v1:172.21.15.89:6789/0 conn(0x7f8aa40d0740 legacy=0x7f8aa4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:27.832 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa9906700 1 -- 172.21.15.89:0/3955981729 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa406f470 con 0x7f8aa415bf60 2023-11-24T06:18:27.832 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa9906700 1 --2- 172.21.15.89:0/3955981729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa415bf60 0x7f8aa40cc710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009840 tx=0x7f8aa0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a27f3bcb1462dd8d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:27.832 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa8904700 1 -- 172.21.15.89:0/3955981729 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c0019f0 con 0x7f8aa415bf60 2023-11-24T06:18:27.832 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa8904700 1 -- 172.21.15.89:0/3955981729 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa0004030 con 0x7f8aa415bf60 2023-11-24T06:18:27.832 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.826+0000 7f8aa8904700 1 -- 172.21.15.89:0/3955981729 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa00043c0 con 0x7f8aa415bf60 2023-11-24T06:18:27.832 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.827+0000 7f8aabb6a700 1 -- 172.21.15.89:0/3955981729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa415bf60 msgr2=0x7f8aa40cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:27.833 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.827+0000 7f8aabb6a700 1 --2- 172.21.15.89:0/3955981729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa415bf60 0x7f8aa40cc710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009840 tx=0x7f8aa0009b50 comp rx=0 tx=0).stop 2023-11-24T06:18:27.833 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.827+0000 7f8aabb6a700 1 -- 172.21.15.89:0/3955981729 shutdown_connections 2023-11-24T06:18:27.833 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.827+0000 7f8aabb6a700 1 --2- 172.21.15.89:0/3955981729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa415bf60 0x7f8aa40cc710 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:27.833 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.827+0000 7f8aabb6a700 1 -- 172.21.15.89:0/3955981729 >> 172.21.15.89:0/3955981729 conn(0x7f8aa409aa00 msgr2=0x7f8aa409ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:27.833 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.827+0000 7f8aabb6a700 1 -- 172.21.15.89:0/3955981729 shutdown_connections 2023-11-24T06:18:27.833 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.827+0000 7f8aabb6a700 1 -- 172.21.15.89:0/3955981729 wait complete. 2023-11-24T06:18:27.836 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.828+0000 7f8aabb6a700 1 Processor -- start 2023-11-24T06:18:27.836 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.828+0000 7f8aabb6a700 1 -- start start 2023-11-24T06:18:27.836 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aabb6a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8aa40d02c0 0x7f8aa4155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:27.836 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aabb6a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa40d0740 0x7f8aa40caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:27.836 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aabb6a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aa415bf60 0x7f8aa41e5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:27.836 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aabb6a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa4155880 con 0x7f8aa415bf60 2023-11-24T06:18:27.836 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aabb6a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa406bee0 con 0x7f8aa40d0740 2023-11-24T06:18:27.836 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aabb6a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa406c1c0 con 0x7f8aa40d02c0 2023-11-24T06:18:27.837 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aaa107700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aa415bf60 0x7f8aa41e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:27.837 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aaa107700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aa415bf60 0x7f8aa41e5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34422/0 (socket says 172.21.15.89:34422) 2023-11-24T06:18:27.837 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aa9105700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa40d0740 0x7f8aa40caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:27.837 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aaa107700 1 -- 172.21.15.89:0/1823266167 learned_addr learned my addr 172.21.15.89:0/1823266167 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:27.837 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aa9906700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8aa40d02c0 0x7f8aa4155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:27.837 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aaa107700 1 -- 172.21.15.89:0/1823266167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8aa40d02c0 msgr2=0x7f8aa4155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:27.837 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aaa107700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8aa40d02c0 0x7f8aa4155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:27.838 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aaa107700 1 -- 172.21.15.89:0/1823266167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa40d0740 msgr2=0x7f8aa40caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:27.838 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aaa107700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa40d0740 0x7f8aa40caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:27.838 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.829+0000 7f8aaa107700 1 -- 172.21.15.89:0/1823266167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa00094a0 con 0x7f8aa415bf60 2023-11-24T06:18:27.838 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.830+0000 7f8aaa107700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aa415bf60 0x7f8aa41e5980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c001ba0 tx=0x7f8a9c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:27.838 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.830+0000 7f8a9a7fc700 1 -- 172.21.15.89:0/1823266167 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c00aac0 con 0x7f8aa415bf60 2023-11-24T06:18:27.838 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.830+0000 7f8a9a7fc700 1 -- 172.21.15.89:0/1823266167 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c00ac20 con 0x7f8aa415bf60 2023-11-24T06:18:27.838 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.830+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa406c4a0 con 0x7f8aa415bf60 2023-11-24T06:18:27.838 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.830+0000 7f8a9a7fc700 1 -- 172.21.15.89:0/1823266167 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c0070d0 con 0x7f8aa415bf60 2023-11-24T06:18:27.839 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.830+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa40caf40 con 0x7f8aa415bf60 2023-11-24T06:18:27.839 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.830+0000 7f8a9a7fc700 1 -- 172.21.15.89:0/1823266167 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8a9c007530 con 0x7f8aa415bf60 2023-11-24T06:18:27.839 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.830+0000 7f8a9a7fc700 1 -- 172.21.15.89:0/1823266167 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8a9c012400 con 0x7f8aa415bf60 2023-11-24T06:18:27.839 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.831+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa40d0130 con 0x7f8aa415bf60 2023-11-24T06:18:27.839 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.833+0000 7f8a9a7fc700 1 -- 172.21.15.89:0/1823266167 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f8a9c018070 con 0x7f8aa415bf60 2023-11-24T06:18:27.872 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:27.869+0000 7fab736cf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-24T06:18:27.905 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.902+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8aa4068fb0 con 0x7f8aa415bf60 2023-11-24T06:18:27.906 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.903+0000 7f8a9a7fc700 1 -- 172.21.15.89:0/1823266167 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f8a9c0077e0 con 0x7f8aa415bf60 2023-11-24T06:18:27.906 INFO:teuthology.misc.health.smithi089.stdout: 2023-11-24T06:18:27.906 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":10,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:26.628120+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:27.907 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aa415bf60 msgr2=0x7f8aa41e5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:27.907 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aa415bf60 0x7f8aa41e5980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c001ba0 tx=0x7f8a9c00cf50 comp rx=0 tx=0).stop 2023-11-24T06:18:27.907 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 shutdown_connections 2023-11-24T06:18:27.907 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aa415bf60 0x7f8aa41e5980 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:27.907 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8aa40d02c0 0x7f8aa4155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:27.908 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 --2- 172.21.15.89:0/1823266167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aa40d0740 0x7f8aa40caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:27.908 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 >> 172.21.15.89:0/1823266167 conn(0x7f8aa409aa00 msgr2=0x7f8aa40b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:27.908 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 shutdown_connections 2023-11-24T06:18:27.908 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:27.904+0000 7f8aabb6a700 1 -- 172.21.15.89:0/1823266167 wait complete. 2023-11-24T06:18:27.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-24T06:18:27.948 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:27.946+0000 7f9c13b1a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-24T06:18:28.006 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:28.003+0000 7fab736cf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-24T06:18:28.079 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:28.077+0000 7f9c13b1a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-24T06:18:28.226 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:28.224+0000 7fab736cf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-24T06:18:28.300 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:28.297+0000 7f9c13b1a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-24T06:18:28.381 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:28.378+0000 7fab736cf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-24T06:18:28.458 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:28.455+0000 7f9c13b1a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-24T06:18:28.495 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:28.492+0000 7fab736cf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-24T06:18:28.570 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:28.568+0000 7f9c13b1a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-24T06:18:28.815 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:28.813+0000 7fab736cf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-24T06:18:28.893 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:28.890+0000 7f9c13b1a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-24T06:18:28.914 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:28.912+0000 7fab736cf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-24T06:18:28.994 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:28.992+0000 7f9c13b1a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-24T06:18:29.103 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:29.100+0000 7fab736cf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-24T06:18:29.180 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:29.178+0000 7f9c13b1a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-24T06:18:29.698 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:29.695+0000 7fab736cf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-24T06:18:29.791 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-11-24T06:18:29.788+0000 7f9c13b1a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-24T06:18:30.721 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:30.718+0000 7fab2b624700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-24T06:18:30.721 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:18:30.718+0000 7fab2b624700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-24T06:18:30.788 INFO:tasks.ceph.osd.5.smithi123.stderr:2023-11-24T06:18:30.785+0000 7f8e532e9640 -1 osd.5 0 log_to_monitors true 2023-11-24T06:18:30.797 INFO:tasks.ceph.osd.1.smithi089.stderr:2023-11-24T06:18:30.794+0000 7f6266296640 -1 osd.1 0 log_to_monitors true 2023-11-24T06:18:30.798 INFO:tasks.ceph.osd.7.smithi123.stderr:2023-11-24T06:18:30.795+0000 7f63ef145640 -1 osd.7 0 log_to_monitors true 2023-11-24T06:18:30.818 INFO:tasks.ceph.osd.0.smithi089.stderr:2023-11-24T06:18:30.815+0000 7f38f25b3640 -1 osd.0 0 log_to_monitors true 2023-11-24T06:18:30.833 INFO:tasks.ceph.osd.4.smithi123.stderr:2023-11-24T06:18:30.831+0000 7fc715699640 -1 osd.4 0 log_to_monitors true 2023-11-24T06:18:30.834 INFO:tasks.ceph.osd.2.smithi089.stderr:2023-11-24T06:18:30.832+0000 7f351510d640 -1 osd.2 0 log_to_monitors true 2023-11-24T06:18:30.851 INFO:tasks.ceph.osd.3.smithi089.stderr:2023-11-24T06:18:30.849+0000 7f81cd9be640 -1 osd.3 0 log_to_monitors true 2023-11-24T06:18:30.857 INFO:tasks.ceph.osd.6.smithi123.stderr:2023-11-24T06:18:30.854+0000 7f2cf2bb9640 -1 osd.6 0 log_to_monitors true 2023-11-24T06:18:34.724 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:34.738 INFO:tasks.ceph.osd.5.smithi123.stderr:2023-11-24T06:18:34.735+0000 7f8e46ea0700 -1 osd.5 0 waiting for initial osdmap 2023-11-24T06:18:34.738 INFO:tasks.ceph.osd.7.smithi123.stderr:2023-11-24T06:18:34.735+0000 7f63e44ff700 -1 osd.7 0 waiting for initial osdmap 2023-11-24T06:18:34.750 INFO:tasks.ceph.osd.5.smithi123.stderr:2023-11-24T06:18:34.748+0000 7f8e3f280700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:18:34.752 INFO:tasks.ceph.osd.3.smithi089.stderr:2023-11-24T06:18:34.750+0000 7f81c1575700 -1 osd.3 0 waiting for initial osdmap 2023-11-24T06:18:34.754 INFO:tasks.ceph.osd.7.smithi123.stderr:2023-11-24T06:18:34.752+0000 7f63db0dc700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:18:34.765 INFO:tasks.ceph.osd.2.smithi089.stderr:2023-11-24T06:18:34.763+0000 7f3508cc4700 -1 osd.2 0 waiting for initial osdmap 2023-11-24T06:18:34.766 INFO:tasks.ceph.osd.3.smithi089.stderr:2023-11-24T06:18:34.764+0000 7f81b9955700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:18:34.769 INFO:tasks.ceph.osd.1.smithi089.stderr:2023-11-24T06:18:34.765+0000 7f6259e4d700 -1 osd.1 0 waiting for initial osdmap 2023-11-24T06:18:34.775 INFO:tasks.ceph.osd.6.smithi123.stderr:2023-11-24T06:18:34.773+0000 7f2ce6770700 -1 osd.6 0 waiting for initial osdmap 2023-11-24T06:18:34.776 INFO:tasks.ceph.osd.2.smithi089.stderr:2023-11-24T06:18:34.773+0000 7f35010a4700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:18:34.777 INFO:tasks.ceph.osd.0.smithi089.stderr:2023-11-24T06:18:34.775+0000 7f38e616a700 -1 osd.0 0 waiting for initial osdmap 2023-11-24T06:18:34.778 INFO:tasks.ceph.osd.1.smithi089.stderr:2023-11-24T06:18:34.776+0000 7f625222d700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:18:34.791 INFO:tasks.ceph.osd.0.smithi089.stderr:2023-11-24T06:18:34.789+0000 7f38de54a700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:18:34.792 INFO:tasks.ceph.osd.6.smithi123.stderr:2023-11-24T06:18:34.790+0000 7f2cdeb50700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:18:34.799 INFO:tasks.ceph.osd.4.smithi123.stderr:2023-11-24T06:18:34.796+0000 7fc709250700 -1 osd.4 0 waiting for initial osdmap 2023-11-24T06:18:34.813 INFO:tasks.ceph.osd.4.smithi123.stderr:2023-11-24T06:18:34.811+0000 7fc701630700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:18:34.854 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.851+0000 7f94c64e6700 1 Processor -- start 2023-11-24T06:18:34.855 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.852+0000 7f94c64e6700 1 -- start start 2023-11-24T06:18:34.858 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.852+0000 7f94c64e6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c00b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:34.858 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.852+0000 7f94c64e6700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f94c00aff30 0x7f94c006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:34.858 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.852+0000 7f94c64e6700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c015ec90 con 0x7f94c00b03b0 2023-11-24T06:18:34.858 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.852+0000 7f94c64e6700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c0158cf0 con 0x7f94c00aff30 2023-11-24T06:18:34.858 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.852+0000 7f94c64e6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c015ef70 con 0x7f94c0158910 2023-11-24T06:18:34.858 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bf7fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f94c00aff30 0x7f94c006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:34.858 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bffff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c00b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94c4a83700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f94c00b03b0 0x7f94c0162210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:41852/0 (socket says 172.21.15.89:41852) 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bf7fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f94c00aff30 0x7f94c006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:49638/0 (socket says 172.21.15.89:49638) 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94c4a83700 1 -- 172.21.15.89:0/2551077636 learned_addr learned my addr 172.21.15.89:0/2551077636 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bffff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c00b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:43400/0 (socket says 172.21.15.89:43400) 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bffff700 1 -- 172.21.15.89:0/2551077636 >> v2:172.21.15.123:3300/0 conn(0x7f94c00aff30 msgr2=0x7f94c006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bffff700 1 --2- 172.21.15.89:0/2551077636 >> v2:172.21.15.123:3300/0 conn(0x7f94c00aff30 0x7f94c006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bffff700 1 -- 172.21.15.89:0/2551077636 >> v1:172.21.15.123:6789/0 conn(0x7f94c00b03b0 legacy=0x7f94c0162210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94beffd700 1 -- 172.21.15.89:0/2551077636 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56587892 0 0) 0x7f94c015ec90 con 0x7f94c00b03b0 2023-11-24T06:18:34.859 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bffff700 1 -- 172.21.15.89:0/2551077636 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c015f250 con 0x7f94c0158910 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bf7fe700 1 --2- 172.21.15.89:0/2551077636 >> v2:172.21.15.123:3300/0 conn(0x7f94c00aff30 0x7f94c006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94bffff700 1 --2- 172.21.15.89:0/2551077636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c00b48e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009840 tx=0x7f94b0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b570b6551ff4e12 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94beffd700 1 -- 172.21.15.89:0/2551077636 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0011830 con 0x7f94c0158910 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94beffd700 1 -- 172.21.15.89:0/2551077636 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0011990 con 0x7f94c0158910 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.853+0000 7f94beffd700 1 -- 172.21.15.89:0/2551077636 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0011d20 con 0x7f94c0158910 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.854+0000 7f94c64e6700 1 -- 172.21.15.89:0/2551077636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 msgr2=0x7f94c00b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.854+0000 7f94c64e6700 1 --2- 172.21.15.89:0/2551077636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c00b48e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009840 tx=0x7f94b0009b50 comp rx=0 tx=0).stop 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.854+0000 7f94c64e6700 1 -- 172.21.15.89:0/2551077636 shutdown_connections 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.854+0000 7f94c64e6700 1 --2- 172.21.15.89:0/2551077636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c00b48e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:34.860 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.854+0000 7f94c64e6700 1 --2- 172.21.15.89:0/2551077636 >> v2:172.21.15.123:3300/0 conn(0x7f94c00aff30 0x7f94c006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:34.861 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.854+0000 7f94c64e6700 1 -- 172.21.15.89:0/2551077636 >> 172.21.15.89:0/2551077636 conn(0x7f94c0071ae0 msgr2=0x7f94c0071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:34.861 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.854+0000 7f94c64e6700 1 -- 172.21.15.89:0/2551077636 shutdown_connections 2023-11-24T06:18:34.861 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.854+0000 7f94c64e6700 1 -- 172.21.15.89:0/2551077636 wait complete. 2023-11-24T06:18:34.861 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.855+0000 7f94c64e6700 1 Processor -- start 2023-11-24T06:18:34.861 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.855+0000 7f94c64e6700 1 -- start start 2023-11-24T06:18:34.864 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94c64e6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94c00aff30 0x7f94c00ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:34.864 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94c64e6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94c00b03b0 0x7f94c014b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:34.864 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94c64e6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c01f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:34.864 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94c64e6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c01f73e0 con 0x7f94c00b03b0 2023-11-24T06:18:34.865 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94c64e6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c0156520 con 0x7f94c00aff30 2023-11-24T06:18:34.865 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94c64e6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c00aeda0 con 0x7f94c0158910 2023-11-24T06:18:34.865 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94c00b03b0 0x7f94c014b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:34.865 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94c00b03b0 0x7f94c014b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34532/0 (socket says 172.21.15.89:34532) 2023-11-24T06:18:34.865 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 -- 172.21.15.89:0/3255261349 learned_addr learned my addr 172.21.15.89:0/3255261349 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:34.865 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bffff700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94c00aff30 0x7f94c00ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:34.865 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94c4a83700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c01f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:34.865 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 -- 172.21.15.89:0/3255261349 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 msgr2=0x7f94c01f7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c01f7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 -- 172.21.15.89:0/3255261349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94c00aff30 msgr2=0x7f94c00ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94c00aff30 0x7f94c00ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 -- 172.21.15.89:0/3255261349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b40094a0 con 0x7f94c00b03b0 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.856+0000 7f94bf7fe700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94c00b03b0 0x7f94c014b7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f94b400ec10 tx=0x7f94b4009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.857+0000 7f94bcff9700 1 -- 172.21.15.89:0/3255261349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b400c8e0 con 0x7f94c00b03b0 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.857+0000 7f94bcff9700 1 -- 172.21.15.89:0/3255261349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b400ca40 con 0x7f94c00b03b0 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.857+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b00094a0 con 0x7f94c00b03b0 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.857+0000 7f94bcff9700 1 -- 172.21.15.89:0/3255261349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b400cdd0 con 0x7f94c00b03b0 2023-11-24T06:18:34.866 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.857+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c014bcf0 con 0x7f94c00b03b0 2023-11-24T06:18:34.867 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.858+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c0158470 con 0x7f94c00b03b0 2023-11-24T06:18:34.867 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.858+0000 7f94bcff9700 1 -- 172.21.15.89:0/3255261349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f94b4013470 con 0x7f94c00b03b0 2023-11-24T06:18:34.867 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.858+0000 7f94bcff9700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94ac06ae40 0x7f94ac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:34.867 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.859+0000 7f94bcff9700 1 -- 172.21.15.89:0/3255261349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f94b40888a0 con 0x7f94c00b03b0 2023-11-24T06:18:34.867 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.859+0000 7f94bffff700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94ac06ae40 0x7f94ac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:34.867 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.859+0000 7f94bffff700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94ac06ae40 0x7f94ac06d300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94b00090b0 tx=0x7f94b0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:34.868 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:34.863+0000 7f94bcff9700 1 -- 172.21.15.89:0/3255261349 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94b40547c0 con 0x7f94c00b03b0 2023-11-24T06:18:35.059 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.056+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94c0068fb0 con 0x7f94c00b03b0 2023-11-24T06:18:35.059 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.057+0000 7f94bcff9700 1 -- 172.21.15.89:0/3255261349 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f94b4017090 con 0x7f94c00b03b0 2023-11-24T06:18:35.060 INFO:teuthology.misc.health.smithi089.stdout: 2023-11-24T06:18:35.060 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":12,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:32.722884+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:35.062 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94ac06ae40 msgr2=0x7f94ac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:35.062 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94ac06ae40 0x7f94ac06d300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94b00090b0 tx=0x7f94b0008030 comp rx=0 tx=0).stop 2023-11-24T06:18:35.062 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94c00b03b0 msgr2=0x7f94c014b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:35.062 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94c00b03b0 0x7f94c014b7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f94b400ec10 tx=0x7f94b4009e20 comp rx=0 tx=0).stop 2023-11-24T06:18:35.062 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 shutdown_connections 2023-11-24T06:18:35.062 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94c0158910 0x7f94c01f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:35.063 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94ac06ae40 0x7f94ac06d300 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:35.063 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94c00aff30 0x7f94c00ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:35.063 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 --2- 172.21.15.89:0/3255261349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94c00b03b0 0x7f94c014b7b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:35.063 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 >> 172.21.15.89:0/3255261349 conn(0x7f94c0071ae0 msgr2=0x7f94c0071ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:35.063 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 shutdown_connections 2023-11-24T06:18:35.063 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:35.059+0000 7f94c64e6700 1 -- 172.21.15.89:0/3255261349 wait complete. 2023-11-24T06:18:35.073 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-24T06:18:41.880 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:42.002 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:41.999+0000 7f6294418700 1 Processor -- start 2023-11-24T06:18:42.002 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:41.999+0000 7f6294418700 1 -- start start 2023-11-24T06:18:42.004 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f6294418700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f628c0d45d0 0x7f628c0b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.004 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f6294418700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f628c0d39d0 0x7f628c0b7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.004 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f6294418700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0d2930 con 0x7f628c15e140 2023-11-24T06:18:42.005 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f6294418700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c15e520 con 0x7f628c0d39d0 2023-11-24T06:18:42.005 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f6294418700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c0d2c10 con 0x7f628c0d45d0 2023-11-24T06:18:42.005 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62929b5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f628c0d39d0 0x7f628c0b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.005 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62929b5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f628c0d39d0 0x7f628c0b7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34762/0 (socket says 172.21.15.89:34762) 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62929b5700 1 -- 172.21.15.89:0/2852256125 learned_addr learned my addr 172.21.15.89:0/2852256125 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62919b3700 1 --2- 172.21.15.89:0/2852256125 >> v2:172.21.15.123:3300/0 conn(0x7f628c0d45d0 0x7f628c0b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62929b5700 1 -- 172.21.15.89:0/2852256125 >> v2:172.21.15.123:3300/0 conn(0x7f628c0d45d0 msgr2=0x7f628c0b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62929b5700 1 --2- 172.21.15.89:0/2852256125 >> v2:172.21.15.123:3300/0 conn(0x7f628c0d45d0 0x7f628c0b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62929b5700 1 -- 172.21.15.89:0/2852256125 >> v1:172.21.15.123:6789/0 conn(0x7f628c15e140 legacy=0x7f628c0d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62929b5700 1 -- 172.21.15.89:0/2852256125 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c0d2ef0 con 0x7f628c0d39d0 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.000+0000 7f62929b5700 1 --2- 172.21.15.89:0/2852256125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c0d39d0 0x7f628c0b7610 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f628c0d4b20 tx=0x7f628400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7ea198534d2615 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f62911b2700 1 -- 172.21.15.89:0/2852256125 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628400e070 con 0x7f628c0d39d0 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f62911b2700 1 -- 172.21.15.89:0/2852256125 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628400b9c0 con 0x7f628c0d39d0 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f62911b2700 1 -- 172.21.15.89:0/2852256125 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628400bd70 con 0x7f628c0d39d0 2023-11-24T06:18:42.006 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f6294418700 1 -- 172.21.15.89:0/2852256125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c0d39d0 msgr2=0x7f628c0b7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.007 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f6294418700 1 --2- 172.21.15.89:0/2852256125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c0d39d0 0x7f628c0b7610 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f628c0d4b20 tx=0x7f628400b330 comp rx=0 tx=0).stop 2023-11-24T06:18:42.007 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f6294418700 1 -- 172.21.15.89:0/2852256125 shutdown_connections 2023-11-24T06:18:42.007 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f6294418700 1 --2- 172.21.15.89:0/2852256125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c0d39d0 0x7f628c0b7610 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.007 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f6294418700 1 --2- 172.21.15.89:0/2852256125 >> v2:172.21.15.123:3300/0 conn(0x7f628c0d45d0 0x7f628c0b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.007 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f6294418700 1 -- 172.21.15.89:0/2852256125 >> 172.21.15.89:0/2852256125 conn(0x7f628c06b410 msgr2=0x7f628c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:42.007 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f6294418700 1 -- 172.21.15.89:0/2852256125 shutdown_connections 2023-11-24T06:18:42.007 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.001+0000 7f6294418700 1 -- 172.21.15.89:0/2852256125 wait complete. 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.002+0000 7f6294418700 1 Processor -- start 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.002+0000 7f6294418700 1 -- start start 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f6294418700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f628c0d39d0 0x7f628c159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f6294418700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f628c0d45d0 0x7f628c15bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f6294418700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c15e140 0x7f628c14e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f6294418700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0cb070 con 0x7f628c15e140 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f6294418700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c158b30 con 0x7f628c0d45d0 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f6294418700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c158e10 con 0x7f628c0d39d0 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f62929b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c15e140 0x7f628c14e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f62929b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c15e140 0x7f628c14e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34768/0 (socket says 172.21.15.89:34768) 2023-11-24T06:18:42.010 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f62929b5700 1 -- 172.21.15.89:0/51850387 learned_addr learned my addr 172.21.15.89:0/51850387 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:42.011 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f62919b3700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f628c0d45d0 0x7f628c15bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.011 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.003+0000 7f62921b4700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f628c0d39d0 0x7f628c159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.011 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f62929b5700 1 -- 172.21.15.89:0/51850387 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f628c0d39d0 msgr2=0x7f628c159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.011 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f62929b5700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f628c0d39d0 0x7f628c159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.011 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f62929b5700 1 -- 172.21.15.89:0/51850387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f628c0d45d0 msgr2=0x7f628c15bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.011 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f62929b5700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f628c0d45d0 0x7f628c15bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.011 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f62929b5700 1 -- 172.21.15.89:0/51850387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6284009b30 con 0x7f628c15e140 2023-11-24T06:18:42.011 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f62929b5700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c15e140 0x7f628c14e090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6284012000 tx=0x7f6284004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f6282ffd700 1 -- 172.21.15.89:0/51850387 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284004880 con 0x7f628c15e140 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f6282ffd700 1 -- 172.21.15.89:0/51850387 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62840099b0 con 0x7f628c15e140 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c1590f0 con 0x7f628c15e140 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f6282ffd700 1 -- 172.21.15.89:0/51850387 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628401c680 con 0x7f628c15e140 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.004+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c1596b0 con 0x7f628c15e140 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.005+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c0d40d0 con 0x7f628c15e140 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.005+0000 7f6282ffd700 1 -- 172.21.15.89:0/51850387 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f628401c7e0 con 0x7f628c15e140 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.005+0000 7f6282ffd700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f627006aed0 0x7f627006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.006+0000 7f6282ffd700 1 -- 172.21.15.89:0/51850387 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f628408aa80 con 0x7f628c15e140 2023-11-24T06:18:42.012 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.006+0000 7f62921b4700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f627006aed0 0x7f627006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.013 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.006+0000 7f62921b4700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f627006aed0 0x7f627006d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6288007fd0 tx=0x7f6288009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.013 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.010+0000 7f6282ffd700 1 -- 172.21.15.89:0/51850387 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6284055f10 con 0x7f628c15e140 2023-11-24T06:18:42.209 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.206+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f628c0d1bb0 con 0x7f628c15e140 2023-11-24T06:18:42.210 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.207+0000 7f6282ffd700 1 -- 172.21.15.89:0/51850387 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12814 (secure 0 0 0) 0x7f6284059530 con 0x7f628c15e140 2023-11-24T06:18:42.210 INFO:teuthology.misc.health.smithi089.stdout: 2023-11-24T06:18:42.210 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":17,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:39.819767+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f627006aed0 msgr2=0x7f627006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f627006aed0 0x7f627006d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6288007fd0 tx=0x7f6288009040 comp rx=0 tx=0).stop 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c15e140 msgr2=0x7f628c14e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c15e140 0x7f628c14e090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6284012000 tx=0x7f6284004ad0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 shutdown_connections 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f628c15e140 0x7f628c14e090 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f628c0d39d0 0x7f628c159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f627006aed0 0x7f627006d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.213 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.209+0000 7f6294418700 1 --2- 172.21.15.89:0/51850387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f628c0d45d0 0x7f628c15bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.214 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.210+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 >> 172.21.15.89:0/51850387 conn(0x7f628c06b410 msgr2=0x7f628c0d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:42.214 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.210+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 shutdown_connections 2023-11-24T06:18:42.214 INFO:teuthology.misc.health.smithi089.stderr:2023-11-24T06:18:42.210+0000 7f6294418700 1 -- 172.21.15.89:0/51850387 wait complete. 2023-11-24T06:18:42.224 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-24T06:18:42.224 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-24T06:18:42.224 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T06:18:42.225 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T06:18:42.227 INFO:tasks.ceph.mds.a:Started 2023-11-24T06:18:42.227 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T06:18:42.227 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T06:18:42.229 INFO:tasks.ceph.mds.c:Started 2023-11-24T06:18:42.229 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T06:18:42.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T06:18:42.232 INFO:tasks.ceph.mds.b:Started 2023-11-24T06:18:42.232 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T06:18:42.232 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T06:18:42.234 INFO:tasks.ceph.mds.d:Started 2023-11-24T06:18:42.234 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-24T06:18:42.235 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:42.306 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2023-11-24T06:18:42.307 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-11-24T06:18:42.323 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2023-11-24T06:18:42.323 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2023-11-24T06:18:42.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.358+0000 7fd85cf9f700 1 Processor -- start 2023-11-24T06:18:42.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.358+0000 7fd85cf9f700 1 -- start start 2023-11-24T06:18:42.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.359+0000 7fd85cf9f700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd85815b200 0x7fd8580b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.359+0000 7fd85cf9f700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8580cfd40 con 0x7fd8580b1a70 2023-11-24T06:18:42.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.359+0000 7fd85cf9f700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8580affa0 con 0x7fd85815ae20 2023-11-24T06:18:42.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.359+0000 7fd85cf9f700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8580d0020 con 0x7fd85815b200 2023-11-24T06:18:42.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.359+0000 7fd855d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd85815b200 0x7fd8580b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd855d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd85815b200 0x7fd8580b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34798/0 (socket says 172.21.15.89:34798) 2023-11-24T06:18:42.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd855d9b700 1 -- 172.21.15.89:0/4136569823 learned_addr learned my addr 172.21.15.89:0/4136569823 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:42.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd856d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd8580b1a70 0x7fd858164b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38304/0 (socket says 172.21.15.89:38304) 2023-11-24T06:18:42.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd855d9b700 1 -- 172.21.15.89:0/4136569823 >> v1:172.21.15.123:6789/0 conn(0x7fd85815ae20 legacy=0x7fd8580b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd855d9b700 1 -- 172.21.15.89:0/4136569823 >> v1:172.21.15.89:6789/0 conn(0x7fd8580b1a70 legacy=0x7fd858164b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd855d9b700 1 -- 172.21.15.89:0/4136569823 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8580b0280 con 0x7fd85815b200 2023-11-24T06:18:42.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd85559a700 1 -- 172.21.15.89:0/4136569823 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809433510 0 0) 0x7fd8580cfd40 con 0x7fd8580b1a70 2023-11-24T06:18:42.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd855d9b700 1 --2- 172.21.15.89:0/4136569823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd85815b200 0x7fd8580b29a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd84000bb90 tx=0x7fd84000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63967655c80da39c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd85559a700 1 -- 172.21.15.89:0/4136569823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84000dd50 con 0x7fd85815b200 2023-11-24T06:18:42.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd85559a700 1 -- 172.21.15.89:0/4136569823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd840010470 con 0x7fd85815b200 2023-11-24T06:18:42.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.360+0000 7fd85559a700 1 -- 172.21.15.89:0/4136569823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840010800 con 0x7fd85815b200 2023-11-24T06:18:42.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.361+0000 7fd85cf9f700 1 -- 172.21.15.89:0/4136569823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd85815b200 msgr2=0x7fd8580b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.361+0000 7fd85cf9f700 1 --2- 172.21.15.89:0/4136569823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd85815b200 0x7fd8580b29a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd84000bb90 tx=0x7fd84000bea0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.361+0000 7fd85cf9f700 1 -- 172.21.15.89:0/4136569823 shutdown_connections 2023-11-24T06:18:42.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.361+0000 7fd85cf9f700 1 --2- 172.21.15.89:0/4136569823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd85815b200 0x7fd8580b29a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.361+0000 7fd85cf9f700 1 -- 172.21.15.89:0/4136569823 >> 172.21.15.89:0/4136569823 conn(0x7fd8580b57b0 msgr2=0x7fd8580b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:42.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.361+0000 7fd85cf9f700 1 -- 172.21.15.89:0/4136569823 shutdown_connections 2023-11-24T06:18:42.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.361+0000 7fd85cf9f700 1 -- 172.21.15.89:0/4136569823 wait complete. 2023-11-24T06:18:42.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.362+0000 7fd85cf9f700 1 Processor -- start 2023-11-24T06:18:42.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.362+0000 7fd85cf9f700 1 -- start start 2023-11-24T06:18:42.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85cf9f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8580b1a70 0x7fd85814baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85cf9f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd85815ae20 0x7fd85815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85cf9f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd85815b200 0x7fd85815e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85cf9f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85815e540 con 0x7fd8580b1a70 2023-11-24T06:18:42.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85cf9f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8580ae990 con 0x7fd85815b200 2023-11-24T06:18:42.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85cf9f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8580cb8a0 con 0x7fd85815ae20 2023-11-24T06:18:42.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8580b1a70 0x7fd85814baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8580b1a70 0x7fd85814baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34802/0 (socket says 172.21.15.89:34802) 2023-11-24T06:18:42.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.363+0000 7fd85659c700 1 -- 172.21.15.89:0/3962803100 learned_addr learned my addr 172.21.15.89:0/3962803100 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:42.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd855d9b700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd85815ae20 0x7fd85815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd856d9d700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd85815b200 0x7fd85815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd85659c700 1 -- 172.21.15.89:0/3962803100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd85815ae20 msgr2=0x7fd85815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd85659c700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd85815ae20 0x7fd85815dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd85659c700 1 -- 172.21.15.89:0/3962803100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd85815b200 msgr2=0x7fd85815e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd85659c700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd85815b200 0x7fd85815e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd85659c700 1 -- 172.21.15.89:0/3962803100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84000b7f0 con 0x7fd8580b1a70 2023-11-24T06:18:42.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd85659c700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8580b1a70 0x7fd85814baa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd848001ed0 tx=0x7fd8480021e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd84effd700 1 -- 172.21.15.89:0/3962803100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848003c00 con 0x7fd8580b1a70 2023-11-24T06:18:42.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd84effd700 1 -- 172.21.15.89:0/3962803100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd848003d60 con 0x7fd8580b1a70 2023-11-24T06:18:42.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8580cbb80 con 0x7fd8580b1a70 2023-11-24T06:18:42.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd84effd700 1 -- 172.21.15.89:0/3962803100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84800ab40 con 0x7fd8580b1a70 2023-11-24T06:18:42.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.364+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8580adc80 con 0x7fd8580b1a70 2023-11-24T06:18:42.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.365+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8580cc1d0 con 0x7fd8580b1a70 2023-11-24T06:18:42.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.365+0000 7fd84effd700 1 -- 172.21.15.89:0/3962803100 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd84800aca0 con 0x7fd8580b1a70 2023-11-24T06:18:42.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.366+0000 7fd84effd700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd83806aed0 0x7fd83806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.366+0000 7fd84effd700 1 -- 172.21.15.89:0/3962803100 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd848089250 con 0x7fd8580b1a70 2023-11-24T06:18:42.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.366+0000 7fd855d9b700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd83806aed0 0x7fd83806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.366+0000 7fd855d9b700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd83806aed0 0x7fd83806d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd84000bb90 tx=0x7fd840013000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.370+0000 7fd84effd700 1 -- 172.21.15.89:0/3962803100 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8480546e0 con 0x7fd8580b1a70 2023-11-24T06:18:42.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.561+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd858067f60 con 0x7fd8580b1a70 2023-11-24T06:18:42.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.562+0000 7fd84effd700 1 -- 172.21.15.89:0/3962803100 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12814 (secure 0 0 0) 0x7fd848057d00 con 0x7fd8580b1a70 2023-11-24T06:18:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:42.565 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:39.819767+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:42.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd83806aed0 msgr2=0x7fd83806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd83806aed0 0x7fd83806d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd84000bb90 tx=0x7fd840013000 comp rx=0 tx=0).stop 2023-11-24T06:18:42.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8580b1a70 msgr2=0x7fd85814baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8580b1a70 0x7fd85814baa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd848001ed0 tx=0x7fd8480021e0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 shutdown_connections 2023-11-24T06:18:42.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd83806aed0 0x7fd83806d390 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd85815b200 0x7fd85815e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8580b1a70 0x7fd85814baa0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 --2- 172.21.15.89:0/3962803100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd85815ae20 0x7fd85815dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 >> 172.21.15.89:0/3962803100 conn(0x7fd8580b57b0 msgr2=0x7fd85806d470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:42.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 shutdown_connections 2023-11-24T06:18:42.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.564+0000 7fd85cf9f700 1 -- 172.21.15.89:0/3962803100 wait complete. 2023-11-24T06:18:42.579 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:18:42.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:18:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.700+0000 7f8eb2aac700 1 Processor -- start 2023-11-24T06:18:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.700+0000 7f8eb2aac700 1 -- start start 2023-11-24T06:18:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb2aac700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8eac0c5810 0x7f8eac165250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb2aac700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac155010 con 0x7f8eac0d0850 2023-11-24T06:18:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb2aac700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac06d4a0 con 0x7f8eac06ee90 2023-11-24T06:18:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb2aac700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac0d0c30 con 0x7f8eac0c5810 2023-11-24T06:18:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb0848700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8eac0d0850 0x7f8eac15b550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38314/0 (socket says 172.21.15.89:38314) 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb0848700 1 -- 172.21.15.89:0/575162347 learned_addr learned my addr 172.21.15.89:0/575162347 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 --2- 172.21.15.89:0/575162347 >> v2:172.21.15.123:3300/0 conn(0x7f8eac0c5810 0x7f8eac165250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945584996 0 0) 0x7f8eac155010 con 0x7f8eac0d0850 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9c0035b0 con 0x7f8eac0d0850 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798165470 0 0) 0x7f8e9c0035b0 con 0x7f8eac0d0850 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 >> v2:172.21.15.123:3300/0 conn(0x7f8eac0c5810 msgr2=0x7f8eac165250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 --2- 172.21.15.89:0/575162347 >> v2:172.21.15.123:3300/0 conn(0x7f8eac0c5810 0x7f8eac165250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 >> v1:172.21.15.123:6789/0 conn(0x7f8eac06ee90 legacy=0x7f8eac0c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac06d700 con 0x7f8eac0d0850 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f8ea0001670 con 0x7f8eac0d0850 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 >> v1:172.21.15.89:6789/0 conn(0x7f8eac0d0850 legacy=0x7f8eac15b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c003a00 0x7f8e9c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e9c004850 0x7f8e9c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e9c005210 0x7f8e9c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0d0c30 con 0x7f8e9c005210 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c00c1b0 con 0x7f8e9c003a00 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c00c370 con 0x7f8e9c004850 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 --2- 172.21.15.89:0/575162347 >> v2:172.21.15.123:3300/0 conn(0x7f8eac0c5810 0x7f8eac165250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e9c005210 0x7f8e9c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eabfff700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e9c004850 0x7f8e9c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb0848700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c003a00 0x7f8e9c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 -- 172.21.15.89:0/575162347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e9c004850 msgr2=0x7f8e9c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e9c004850 0x7f8e9c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 -- 172.21.15.89:0/575162347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c003a00 msgr2=0x7f8e9c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c003a00 0x7f8e9c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 -- 172.21.15.89:0/575162347 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c0035b0 con 0x7f8e9c005210 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 -- 172.21.15.89:0/575162347 reap_dead start 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eb1049700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e9c005210 0x7f8e9c00ba10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400e350 tx=0x7f8e9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6968767ced8595 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea400ee10 con 0x7f8e9c005210 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea400f040 con 0x7f8e9c005210 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.701+0000 7f8eab7fe700 1 -- 172.21.15.89:0/575162347 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4003c60 con 0x7f8e9c005210 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.702+0000 7f8eb2aac700 1 -- 172.21.15.89:0/575162347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e9c005210 msgr2=0x7f8e9c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.702+0000 7f8eb2aac700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e9c005210 0x7f8e9c00ba10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400e350 tx=0x7f8e9c005880 comp rx=0 tx=0).stop 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.702+0000 7f8eb2aac700 1 -- 172.21.15.89:0/575162347 shutdown_connections 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.702+0000 7f8eb2aac700 1 --2- 172.21.15.89:0/575162347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e9c005210 0x7f8e9c00ba10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.702+0000 7f8eb2aac700 1 -- 172.21.15.89:0/575162347 >> 172.21.15.89:0/575162347 conn(0x7f8eac0726e0 msgr2=0x7f8eac072af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.702+0000 7f8eb2aac700 1 -- 172.21.15.89:0/575162347 shutdown_connections 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.702+0000 7f8eb2aac700 1 -- 172.21.15.89:0/575162347 wait complete. 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.703+0000 7f8eb2aac700 1 Processor -- start 2023-11-24T06:18:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.703+0000 7f8eb2aac700 1 -- start start 2023-11-24T06:18:42.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eb2aac700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c005210 0x7f8eac157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eb2aac700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eac157670 0x7f8eac09b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eb2aac700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eac159bf0 0x7f8eac09b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eb2aac700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0b4310 con 0x7f8eac157670 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eb2aac700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0b5e00 con 0x7f8e9c005210 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eb2aac700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac156ca0 con 0x7f8eac159bf0 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eabfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eac157670 0x7f8eac09b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eabfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eac157670 0x7f8eac09b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34824/0 (socket says 172.21.15.89:34824) 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eabfff700 1 -- 172.21.15.89:0/3739541899 learned_addr learned my addr 172.21.15.89:0/3739541899 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.704+0000 7f8eb0848700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c005210 0x7f8eac157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eb1049700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eac159bf0 0x7f8eac09b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eabfff700 1 -- 172.21.15.89:0/3739541899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eac159bf0 msgr2=0x7f8eac09b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eabfff700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eac159bf0 0x7f8eac09b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eabfff700 1 -- 172.21.15.89:0/3739541899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c005210 msgr2=0x7f8eac157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eabfff700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c005210 0x7f8eac157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eabfff700 1 -- 172.21.15.89:0/3739541899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac0b5840 con 0x7f8eac157670 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eabfff700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eac157670 0x7f8eac09b3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800f880 tx=0x7f8e9800fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8ea97fa700 1 -- 172.21.15.89:0/3739541899 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e980052e0 con 0x7f8eac157670 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8ea97fa700 1 -- 172.21.15.89:0/3739541899 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98005440 con 0x7f8eac157670 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac0b5b20 con 0x7f8eac157670 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8ea97fa700 1 -- 172.21.15.89:0/3739541899 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e980038c0 con 0x7f8eac157670 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.705+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac0b3490 con 0x7f8eac157670 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.706+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac0c6120 con 0x7f8eac157670 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.706+0000 7f8ea97fa700 1 -- 172.21.15.89:0/3739541899 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8e98003a20 con 0x7f8eac157670 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.707+0000 7f8ea97fa700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8e9006aed0 0x7f8e9006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.707+0000 7f8ea97fa700 1 -- 172.21.15.89:0/3739541899 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e98088d80 con 0x7f8eac157670 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.707+0000 7f8eb0848700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8e9006aed0 0x7f8e9006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:42.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.707+0000 7f8eb0848700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8e9006aed0 0x7f8e9006d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0001050 tx=0x7f8ea0001c30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:42.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.712+0000 7f8ea97fa700 1 -- 172.21.15.89:0/3739541899 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e98054230 con 0x7f8eac157670 2023-11-24T06:18:42.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.902+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8eac0b1010 con 0x7f8eac157670 2023-11-24T06:18:42.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.902+0000 7f8ea97fa700 1 -- 172.21.15.89:0/3739541899 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e98057850 con 0x7f8eac157670 2023-11-24T06:18:42.905 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8e9006aed0 msgr2=0x7f8e9006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8e9006aed0 0x7f8e9006d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0001050 tx=0x7f8ea0001c30 comp rx=0 tx=0).stop 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eac157670 msgr2=0x7f8eac09b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eac157670 0x7f8eac09b3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800f880 tx=0x7f8e9800fb90 comp rx=0 tx=0).stop 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 shutdown_connections 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eac159bf0 0x7f8eac09b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8e9006aed0 0x7f8e9006d390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e9c005210 0x7f8eac157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 --2- 172.21.15.89:0/3739541899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eac157670 0x7f8eac09b3c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:42.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 >> 172.21.15.89:0/3739541899 conn(0x7f8eac0726e0 msgr2=0x7f8eac061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:42.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.904+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 shutdown_connections 2023-11-24T06:18:42.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:42.905+0000 7f8eb2aac700 1 -- 172.21.15.89:0/3739541899 wait complete. 2023-11-24T06:18:42.919 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:18:43.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.036+0000 7f4bc0d04700 1 Processor -- start 2023-11-24T06:18:43.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.036+0000 7f4bc0d04700 1 -- start start 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bc0d04700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4bbc0b3370 0x7f4bbc1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bc0d04700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4bbc0d1c70 0x7f4bbc0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bc0d04700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0b22d0 con 0x7f4bbc15bdd0 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bc0d04700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc0b2f50 con 0x7f4bbc0b3370 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bc0d04700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc0d2640 con 0x7f4bbc0d1c70 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bb9d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4bbc0b3370 0x7f4bbc1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bb9d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4bbc0b3370 0x7f4bbc1604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34832/0 (socket says 172.21.15.89:34832) 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bb9d9b700 1 -- 172.21.15.89:0/3020041655 learned_addr learned my addr 172.21.15.89:0/3020041655 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.037+0000 7f4bbad9d700 1 --2- 172.21.15.89:0/3020041655 >> v2:172.21.15.123:3300/0 conn(0x7f4bbc0d1c70 0x7f4bbc0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:43.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bb9d9b700 1 -- 172.21.15.89:0/3020041655 >> v2:172.21.15.123:3300/0 conn(0x7f4bbc0d1c70 msgr2=0x7f4bbc0d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bb9d9b700 1 --2- 172.21.15.89:0/3020041655 >> v2:172.21.15.123:3300/0 conn(0x7f4bbc0d1c70 0x7f4bbc0d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bb9d9b700 1 -- 172.21.15.89:0/3020041655 >> v1:172.21.15.123:6789/0 conn(0x7f4bbc15bdd0 legacy=0x7f4bbc0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bb9d9b700 1 -- 172.21.15.89:0/3020041655 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0b25b0 con 0x7f4bbc0b3370 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bbad9d700 1 --2- 172.21.15.89:0/3020041655 >> v2:172.21.15.123:3300/0 conn(0x7f4bbc0d1c70 0x7f4bbc0d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bb9d9b700 1 --2- 172.21.15.89:0/3020041655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc1604e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc160a50 tx=0x7f4bac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4a348c3a5776d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bb959a700 1 -- 172.21.15.89:0/3020041655 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac00b870 con 0x7f4bbc0b3370 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bb959a700 1 -- 172.21.15.89:0/3020041655 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac00b9d0 con 0x7f4bbc0b3370 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bb959a700 1 -- 172.21.15.89:0/3020041655 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac00bd60 con 0x7f4bbc0b3370 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bc0d04700 1 -- 172.21.15.89:0/3020041655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 msgr2=0x7f4bbc1604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/3020041655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc1604e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc160a50 tx=0x7f4bac009bd0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bc0d04700 1 -- 172.21.15.89:0/3020041655 shutdown_connections 2023-11-24T06:18:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/3020041655 >> v2:172.21.15.123:3300/0 conn(0x7f4bbc0d1c70 0x7f4bbc0d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/3020041655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc1604e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.038+0000 7f4bc0d04700 1 -- 172.21.15.89:0/3020041655 >> 172.21.15.89:0/3020041655 conn(0x7f4bbc0731a0 msgr2=0x7f4bbc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.039+0000 7f4bc0d04700 1 -- 172.21.15.89:0/3020041655 shutdown_connections 2023-11-24T06:18:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.039+0000 7f4bc0d04700 1 -- 172.21.15.89:0/3020041655 wait complete. 2023-11-24T06:18:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.040+0000 7f4bc0d04700 1 Processor -- start 2023-11-24T06:18:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.040+0000 7f4bc0d04700 1 -- start start 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.040+0000 7f4bc0d04700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc0c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.040+0000 7f4bc0d04700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4bbc0d1c70 0x7f4bbc1e8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bc0d04700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bbc15bdd0 0x7f4bbc1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bc0d04700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0c9390 con 0x7f4bbc0b3370 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bc0d04700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc1e86b0 con 0x7f4bbc15bdd0 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bc0d04700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc1577f0 con 0x7f4bbc0d1c70 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc0c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc0c8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34836/0 (socket says 172.21.15.89:34836) 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 -- 172.21.15.89:0/2700430069 learned_addr learned my addr 172.21.15.89:0/2700430069 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bbad9d700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bbc15bdd0 0x7f4bbc1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bb9d9b700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4bbc0d1c70 0x7f4bbc1e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 -- 172.21.15.89:0/2700430069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4bbc0d1c70 msgr2=0x7f4bbc1e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4bbc0d1c70 0x7f4bbc1e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 -- 172.21.15.89:0/2700430069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bbc15bdd0 msgr2=0x7f4bbc1567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bbc15bdd0 0x7f4bbc1567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 -- 172.21.15.89:0/2700430069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac009550 con 0x7f4bbc0b3370 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bba59c700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc0c8db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000c480 tx=0x7f4bb000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4baaffd700 1 -- 172.21.15.89:0/2700430069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb00052d0 con 0x7f4bbc0b3370 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4baaffd700 1 -- 172.21.15.89:0/2700430069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb0005430 con 0x7f4bbc0b3370 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc145870 con 0x7f4bbc0b3370 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4baaffd700 1 -- 172.21.15.89:0/2700430069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb00038c0 con 0x7f4bbc0b3370 2023-11-24T06:18:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.041+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc0cac90 con 0x7f4bbc0b3370 2023-11-24T06:18:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.042+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc067f60 con 0x7f4bbc0b3370 2023-11-24T06:18:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.043+0000 7f4baaffd700 1 -- 172.21.15.89:0/2700430069 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4bb0003a20 con 0x7f4bbc0b3370 2023-11-24T06:18:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.043+0000 7f4baaffd700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b9c06aed0 0x7f4b9c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.043+0000 7f4bb9d9b700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b9c06aed0 0x7f4b9c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.043+0000 7f4baaffd700 1 -- 172.21.15.89:0/2700430069 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4bb00885e0 con 0x7f4bbc0b3370 2023-11-24T06:18:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.043+0000 7f4bb9d9b700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b9c06aed0 0x7f4b9c06d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4bac000c00 tx=0x7f4bac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.048+0000 7f4baaffd700 1 -- 172.21.15.89:0/2700430069 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4bb0053950 con 0x7f4bbc0b3370 2023-11-24T06:18:43.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.281+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bbc0b7c60 con 0x7f4bbc0b3370 2023-11-24T06:18:43.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.281+0000 7f4baaffd700 1 -- 172.21.15.89:0/2700430069 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f4bb0017090 con 0x7f4bbc0b3370 2023-11-24T06:18:43.285 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:43.285 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-11-24T06:18:43.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.283+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b9c06aed0 msgr2=0x7f4b9c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:43.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.283+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b9c06aed0 0x7f4b9c06d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4bac000c00 tx=0x7f4bac008030 comp rx=0 tx=0).stop 2023-11-24T06:18:43.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.283+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 msgr2=0x7f4bbc0c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:43.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.283+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc0c8db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000c480 tx=0x7f4bb000c790 comp rx=0 tx=0).stop 2023-11-24T06:18:43.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.284+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 shutdown_connections 2023-11-24T06:18:43.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.284+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b9c06aed0 0x7f4b9c06d390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.284+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bbc15bdd0 0x7f4bbc1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.284+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bbc0b3370 0x7f4bbc0c8db0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.284+0000 7f4bc0d04700 1 --2- 172.21.15.89:0/2700430069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4bbc0d1c70 0x7f4bbc1e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:43.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.284+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 >> 172.21.15.89:0/2700430069 conn(0x7f4bbc0731a0 msgr2=0x7f4bbc0b0360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:43.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.284+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 shutdown_connections 2023-11-24T06:18:43.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:43.284+0000 7f4bc0d04700 1 -- 172.21.15.89:0/2700430069 wait complete. 2023-11-24T06:18:43.288 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 2 2023-11-24T06:18:45.300 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:18:45.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.431+0000 7f4b8b367700 1 Processor -- start 2023-11-24T06:18:45.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.431+0000 7f4b8b367700 1 -- start start 2023-11-24T06:18:45.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.432+0000 7f4b8b367700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4b840b49a0 0x7f4b840d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.432+0000 7f4b8b367700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8415c4b0 con 0x7f4b84152cc0 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.432+0000 7f4b8b367700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8415c790 con 0x7f4b8406c710 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.432+0000 7f4b8b367700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8415ca70 con 0x7f4b840b49a0 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.432+0000 7f4b89103700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4b84152cc0 0x7f4b8406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38350/0 (socket says 172.21.15.89:38350) 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.432+0000 7f4b89103700 1 -- 172.21.15.89:0/210363201 learned_addr learned my addr 172.21.15.89:0/210363201 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.432+0000 7f4b89904700 1 --2- 172.21.15.89:0/210363201 >> v2:172.21.15.123:3300/0 conn(0x7f4b840b49a0 0x7f4b840d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.432+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522001622 0 0) 0x7f4b8415c4b0 con 0x7f4b84152cc0 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b700035b0 con 0x7f4b84152cc0 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1733167805 0 0) 0x7f4b700035b0 con 0x7f4b84152cc0 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 >> v2:172.21.15.123:3300/0 conn(0x7f4b840b49a0 msgr2=0x7f4b840d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:45.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 --2- 172.21.15.89:0/210363201 >> v2:172.21.15.123:3300/0 conn(0x7f4b840b49a0 0x7f4b840d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 >> v1:172.21.15.123:6789/0 conn(0x7f4b8406c710 legacy=0x7f4b840cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8415cd50 con 0x7f4b84152cc0 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f4b80001670 con 0x7f4b84152cc0 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 >> v1:172.21.15.89:6789/0 conn(0x7f4b84152cc0 legacy=0x7f4b8406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b70003a00 0x7f4b70003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b70004850 0x7f4b70004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70005210 0x7f4b7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8415ca70 con 0x7f4b70004850 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7000c1b0 con 0x7f4b70005210 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7000c370 con 0x7f4b70003a00 2023-11-24T06:18:45.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b88902700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b70004850 0x7f4b70004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b89103700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b70003a00 0x7f4b70003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b89904700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70005210 0x7f4b7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b88902700 1 -- 172.21.15.89:0/210363201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b70003a00 msgr2=0x7f4b70003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b88902700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b70003a00 0x7f4b70003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b88902700 1 -- 172.21.15.89:0/210363201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70005210 msgr2=0x7f4b7000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b88902700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70005210 0x7f4b7000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b88902700 1 -- 172.21.15.89:0/210363201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700035b0 con 0x7f4b70004850 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b89904700 1 -- 172.21.15.89:0/210363201 reap_dead start 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b88902700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b70004850 0x7f4b70004cd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4b740017c0 tx=0x7f4b70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473e7b32111b8221 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.433+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b740052d0 con 0x7f4b70004850 2023-11-24T06:18:45.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b74005430 con 0x7f4b70004850 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b7bfff700 1 -- 172.21.15.89:0/210363201 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b740038c0 con 0x7f4b70004850 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b8b367700 1 -- 172.21.15.89:0/210363201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b70004850 msgr2=0x7f4b70004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b8b367700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b70004850 0x7f4b70004cd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4b740017c0 tx=0x7f4b70005880 comp rx=0 tx=0).stop 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b8b367700 1 -- 172.21.15.89:0/210363201 shutdown_connections 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b8b367700 1 --2- 172.21.15.89:0/210363201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b70004850 0x7f4b70004cd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b8b367700 1 -- 172.21.15.89:0/210363201 >> 172.21.15.89:0/210363201 conn(0x7f4b84071b90 msgr2=0x7f4b84071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b8b367700 1 -- 172.21.15.89:0/210363201 shutdown_connections 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.434+0000 7f4b8b367700 1 -- 172.21.15.89:0/210363201 wait complete. 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.436+0000 7f4b8b367700 1 Processor -- start 2023-11-24T06:18:45.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.436+0000 7f4b8b367700 1 -- start start 2023-11-24T06:18:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b8b367700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70004850 0x7f4b84159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b8b367700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b8415dda0 0x7f4b8415e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b8b367700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b84148280 0x7f4b84148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b8b367700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84148c40 con 0x7f4b8415dda0 2023-11-24T06:18:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b8b367700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8415e8b0 con 0x7f4b70004850 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b8b367700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8415eb90 con 0x7f4b84148280 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b8415dda0 0x7f4b8415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b8415dda0 0x7f4b8415e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34860/0 (socket says 172.21.15.89:34860) 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 -- 172.21.15.89:0/1007461108 learned_addr learned my addr 172.21.15.89:0/1007461108 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 -- 172.21.15.89:0/1007461108 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b84148280 msgr2=0x7f4b84148700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b84148280 0x7f4b84148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 -- 172.21.15.89:0/1007461108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70004850 msgr2=0x7f4b84159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b89103700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70004850 0x7f4b84159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70004850 0x7f4b84159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 -- 172.21.15.89:0/1007461108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b841574d0 con 0x7f4b8415dda0 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b88902700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b8415dda0 0x7f4b8415e220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4b74015040 tx=0x7f4b74001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b79ffb700 1 -- 172.21.15.89:0/1007461108 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b74001290 con 0x7f4b8415dda0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b79ffb700 1 -- 172.21.15.89:0/1007461108 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b74003e80 con 0x7f4b8415dda0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b841577b0 con 0x7f4b8415dda0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.437+0000 7f4b79ffb700 1 -- 172.21.15.89:0/1007461108 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b74006a80 con 0x7f4b8415dda0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.438+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b84159b80 con 0x7f4b8415dda0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.438+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b840753f0 con 0x7f4b8415dda0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.439+0000 7f4b79ffb700 1 -- 172.21.15.89:0/1007461108 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4b74019040 con 0x7f4b8415dda0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.439+0000 7f4b79ffb700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b6806aed0 0x7f4b6806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.439+0000 7f4b79ffb700 1 -- 172.21.15.89:0/1007461108 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4b74089ca0 con 0x7f4b8415dda0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.439+0000 7f4b89103700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b6806aed0 0x7f4b6806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.440+0000 7f4b89103700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b6806aed0 0x7f4b6806d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4b80001050 tx=0x7f4b80009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.444+0000 7f4b79ffb700 1 -- 172.21.15.89:0/1007461108 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b74055250 con 0x7f4b8415dda0 2023-11-24T06:18:45.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.675+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b84157310 con 0x7f4b8415dda0 2023-11-24T06:18:45.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.675+0000 7f4b79ffb700 1 -- 172.21.15.89:0/1007461108 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3691 (secure 0 0 0) 0x7f4b74058870 con 0x7f4b8415dda0 2023-11-24T06:18:45.677 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:45.678 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.677+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b6806aed0 msgr2=0x7f4b6806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.677+0000 7f4b8b367700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b6806aed0 0x7f4b6806d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4b80001050 tx=0x7f4b80009040 comp rx=0 tx=0).stop 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.677+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b8415dda0 msgr2=0x7f4b8415e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.677+0000 7f4b8b367700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b8415dda0 0x7f4b8415e220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4b74015040 tx=0x7f4b74001120 comp rx=0 tx=0).stop 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.678+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 shutdown_connections 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.678+0000 7f4b8b367700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b84148280 0x7f4b84148700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.678+0000 7f4b8b367700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4b6806aed0 0x7f4b6806d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.678+0000 7f4b8b367700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b70004850 0x7f4b84159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.678+0000 7f4b8b367700 1 --2- 172.21.15.89:0/1007461108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b8415dda0 0x7f4b8415e220 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.678+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 >> 172.21.15.89:0/1007461108 conn(0x7f4b84071b90 msgr2=0x7f4b8415a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:45.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.678+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 shutdown_connections 2023-11-24T06:18:45.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.678+0000 7f4b8b367700 1 -- 172.21.15.89:0/1007461108 wait complete. 2023-11-24T06:18:45.681 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 3 2023-11-24T06:18:45.692 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.810+0000 7fc756065700 1 Processor -- start 2023-11-24T06:18:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.810+0000 7fc756065700 1 -- start start 2023-11-24T06:18:45.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.810+0000 7fc756065700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7501446a0 0x7fc7500b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.810+0000 7fc756065700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc7500adf00 0x7fc75006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.810+0000 7fc756065700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7500af950 con 0x7fc75006c540 2023-11-24T06:18:45.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.810+0000 7fc756065700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc750144a80 con 0x7fc7500adf00 2023-11-24T06:18:45.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.810+0000 7fc756065700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75006c9b0 con 0x7fc7501446a0 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74f7fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7501446a0 0x7fc7500b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74effd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc7500adf00 0x7fc75006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74f7fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7501446a0 0x7fc7500b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:43742/0 (socket says 172.21.15.89:43742) 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74f7fe700 1 -- 172.21.15.89:0/1149999614 learned_addr learned my addr 172.21.15.89:0/1149999614 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74ffff700 1 --1- 172.21.15.89:0/1149999614 >> v1:172.21.15.123:6789/0 conn(0x7fc75006c540 0x7fc75006da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:42196/0 (socket says 172.21.15.89:42196) 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74f7fe700 1 -- 172.21.15.89:0/1149999614 >> v2:172.21.15.123:3300/0 conn(0x7fc7500adf00 msgr2=0x7fc75006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74f7fe700 1 --2- 172.21.15.89:0/1149999614 >> v2:172.21.15.123:3300/0 conn(0x7fc7500adf00 0x7fc75006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74f7fe700 1 -- 172.21.15.89:0/1149999614 >> v1:172.21.15.123:6789/0 conn(0x7fc75006c540 legacy=0x7fc75006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74f7fe700 1 -- 172.21.15.89:0/1149999614 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7500afc30 con 0x7fc7501446a0 2023-11-24T06:18:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74f7fe700 1 --2- 172.21.15.89:0/1149999614 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7501446a0 0x7fc7500b5ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7380098f0 tx=0x7fc738009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e9ae8519dab8bf0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74e7fc700 1 -- 172.21.15.89:0/1149999614 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738011830 con 0x7fc7501446a0 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74e7fc700 1 -- 172.21.15.89:0/1149999614 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc738011990 con 0x7fc7501446a0 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.811+0000 7fc74e7fc700 1 -- 172.21.15.89:0/1149999614 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738011d20 con 0x7fc7501446a0 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.812+0000 7fc756065700 1 -- 172.21.15.89:0/1149999614 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7501446a0 msgr2=0x7fc7500b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.812+0000 7fc756065700 1 --2- 172.21.15.89:0/1149999614 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7501446a0 0x7fc7500b5ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7380098f0 tx=0x7fc738009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.812+0000 7fc756065700 1 -- 172.21.15.89:0/1149999614 shutdown_connections 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.812+0000 7fc756065700 1 --2- 172.21.15.89:0/1149999614 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7501446a0 0x7fc7500b5ae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.812+0000 7fc756065700 1 --2- 172.21.15.89:0/1149999614 >> v2:172.21.15.123:3300/0 conn(0x7fc7500adf00 0x7fc75006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.812+0000 7fc756065700 1 -- 172.21.15.89:0/1149999614 >> 172.21.15.89:0/1149999614 conn(0x7fc750071b90 msgr2=0x7fc750071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.812+0000 7fc756065700 1 -- 172.21.15.89:0/1149999614 shutdown_connections 2023-11-24T06:18:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.812+0000 7fc756065700 1 -- 172.21.15.89:0/1149999614 wait complete. 2023-11-24T06:18:45.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.813+0000 7fc756065700 1 Processor -- start 2023-11-24T06:18:45.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.813+0000 7fc756065700 1 -- start start 2023-11-24T06:18:45.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc756065700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75006c540 0x7fc75014ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc756065700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7500adf00 0x7fc75014d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc756065700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7501446a0 0x7fc7500cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc756065700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75014dce0 con 0x7fc7500adf00 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc756065700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75014dfc0 con 0x7fc7501446a0 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc756065700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75014e2a0 con 0x7fc75006c540 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7500adf00 0x7fc75014d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7500adf00 0x7fc75014d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34874/0 (socket says 172.21.15.89:34874) 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74effd700 1 -- 172.21.15.89:0/129248341 learned_addr learned my addr 172.21.15.89:0/129248341 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74ffff700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7501446a0 0x7fc7500cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74f7fe700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75006c540 0x7fc75014ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74effd700 1 -- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75006c540 msgr2=0x7fc75014ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74effd700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75006c540 0x7fc75014ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74effd700 1 -- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7501446a0 msgr2=0x7fc7500cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74effd700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7501446a0 0x7fc7500cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.814+0000 7fc74effd700 1 -- 172.21.15.89:0/129248341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc738009550 con 0x7fc7500adf00 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.815+0000 7fc74ffff700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7501446a0 0x7fc7500cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.815+0000 7fc74f7fe700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75006c540 0x7fc75014ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.815+0000 7fc74effd700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7500adf00 0x7fc75014d3e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc74000e980 tx=0x7fc74000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.815+0000 7fc747fff700 1 -- 172.21.15.89:0/129248341 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740009710 con 0x7fc7500adf00 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.815+0000 7fc747fff700 1 -- 172.21.15.89:0/129248341 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74000c850 con 0x7fc7500adf00 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.815+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75014e580 con 0x7fc7500adf00 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.815+0000 7fc747fff700 1 -- 172.21.15.89:0/129248341 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74000cc00 con 0x7fc7500adf00 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.815+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7500d0160 con 0x7fc7500adf00 2023-11-24T06:18:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.816+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7500753f0 con 0x7fc7500adf00 2023-11-24T06:18:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.816+0000 7fc747fff700 1 -- 172.21.15.89:0/129248341 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc740013470 con 0x7fc7500adf00 2023-11-24T06:18:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.816+0000 7fc747fff700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc73006aed0 0x7fc73006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.817+0000 7fc747fff700 1 -- 172.21.15.89:0/129248341 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc7400892d0 con 0x7fc7500adf00 2023-11-24T06:18:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.817+0000 7fc74f7fe700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc73006aed0 0x7fc73006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.817+0000 7fc74f7fe700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc73006aed0 0x7fc73006d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc738000c00 tx=0x7fc73801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:45.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:45.821+0000 7fc747fff700 1 -- 172.21.15.89:0/129248341 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc740054820 con 0x7fc7500adf00 2023-11-24T06:18:46.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.014+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc75014eaa0 con 0x7fc7500adf00 2023-11-24T06:18:46.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.014+0000 7fc747fff700 1 -- 172.21.15.89:0/129248341 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12814 (secure 0 0 0) 0x7fc75014eaa0 con 0x7fc7500adf00 2023-11-24T06:18:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:46.017 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:39.819767+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc73006aed0 msgr2=0x7fc73006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc73006aed0 0x7fc73006d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc738000c00 tx=0x7fc73801c040 comp rx=0 tx=0).stop 2023-11-24T06:18:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7500adf00 msgr2=0x7fc75014d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7500adf00 0x7fc75014d3e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc74000e980 tx=0x7fc74000ed40 comp rx=0 tx=0).stop 2023-11-24T06:18:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 shutdown_connections 2023-11-24T06:18:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc73006aed0 0x7fc73006d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7501446a0 0x7fc7500cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75006c540 0x7fc75014ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 --2- 172.21.15.89:0/129248341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7500adf00 0x7fc75014d3e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 >> 172.21.15.89:0/129248341 conn(0x7fc750071b90 msgr2=0x7fc75015dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 shutdown_connections 2023-11-24T06:18:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.017+0000 7fc756065700 1 -- 172.21.15.89:0/129248341 wait complete. 2023-11-24T06:18:46.031 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:18:46.031 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:18:46.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.151+0000 7f8abcb92700 1 Processor -- start 2023-11-24T06:18:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.151+0000 7f8abcb92700 1 -- start start 2023-11-24T06:18:46.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.152+0000 7f8abcb92700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8ab815a620 0x7f8ab80ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.152+0000 7f8abcb92700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab806ecf0 0x7f8ab80b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.152+0000 7f8abcb92700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80b2f20 con 0x7f8ab80cc540 2023-11-24T06:18:46.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.152+0000 7f8abcb92700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab80ce690 con 0x7f8ab815a620 2023-11-24T06:18:46.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.152+0000 7f8abcb92700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab815aa00 con 0x7f8ab806ecf0 2023-11-24T06:18:46.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.152+0000 7f8ab659c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8ab815a620 0x7f8ab80ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab659c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8ab815a620 0x7f8ab80ce150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34880/0 (socket says 172.21.15.89:34880) 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab659c700 1 -- 172.21.15.89:0/3934067905 learned_addr learned my addr 172.21.15.89:0/3934067905 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab6d9d700 1 --2- 172.21.15.89:0/3934067905 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab806ecf0 0x7f8ab80b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab659c700 1 -- 172.21.15.89:0/3934067905 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab806ecf0 msgr2=0x7f8ab80b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab659c700 1 --2- 172.21.15.89:0/3934067905 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab806ecf0 0x7f8ab80b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab659c700 1 -- 172.21.15.89:0/3934067905 >> v1:172.21.15.123:6789/0 conn(0x7f8ab80cc540 legacy=0x7f8ab806e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab659c700 1 -- 172.21.15.89:0/3934067905 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab80ce960 con 0x7f8ab815a620 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab659c700 1 --2- 172.21.15.89:0/3934067905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab815a620 0x7f8ab80ce150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00bb90 tx=0x7f8aac00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285ab007124b23b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab559a700 1 -- 172.21.15.89:0/3934067905 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac00dd50 con 0x7f8ab815a620 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab559a700 1 -- 172.21.15.89:0/3934067905 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac010470 con 0x7f8ab815a620 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.153+0000 7f8ab559a700 1 -- 172.21.15.89:0/3934067905 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac010800 con 0x7f8ab815a620 2023-11-24T06:18:46.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.154+0000 7f8abcb92700 1 -- 172.21.15.89:0/3934067905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab815a620 msgr2=0x7f8ab80ce150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.154+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3934067905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab815a620 0x7f8ab80ce150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00bb90 tx=0x7f8aac00bea0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.154+0000 7f8abcb92700 1 -- 172.21.15.89:0/3934067905 shutdown_connections 2023-11-24T06:18:46.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.154+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3934067905 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab806ecf0 0x7f8ab80b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.154+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3934067905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab815a620 0x7f8ab80ce150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.154+0000 7f8abcb92700 1 -- 172.21.15.89:0/3934067905 >> 172.21.15.89:0/3934067905 conn(0x7f8ab80b2050 msgr2=0x7f8ab80b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:46.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.154+0000 7f8abcb92700 1 -- 172.21.15.89:0/3934067905 shutdown_connections 2023-11-24T06:18:46.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.154+0000 7f8abcb92700 1 -- 172.21.15.89:0/3934067905 wait complete. 2023-11-24T06:18:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.155+0000 7f8abcb92700 1 Processor -- start 2023-11-24T06:18:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.155+0000 7f8abcb92700 1 -- start start 2023-11-24T06:18:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8abcb92700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ab806ecf0 0x7f8ab80ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8abcb92700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab80cc540 0x7f8ab814acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8abcb92700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab815a620 0x7f8ab81543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8abcb92700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab8150dc0 con 0x7f8ab80cc540 2023-11-24T06:18:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8abcb92700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab80ae7c0 con 0x7f8ab806ecf0 2023-11-24T06:18:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8abcb92700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab81501b0 con 0x7f8ab815a620 2023-11-24T06:18:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab80cc540 0x7f8ab814acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab80cc540 0x7f8ab814acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34886/0 (socket says 172.21.15.89:34886) 2023-11-24T06:18:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 -- 172.21.15.89:0/3399479139 learned_addr learned my addr 172.21.15.89:0/3399479139 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab6d9d700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab815a620 0x7f8ab81543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab659c700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ab806ecf0 0x7f8ab80ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 -- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab815a620 msgr2=0x7f8ab81543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab815a620 0x7f8ab81543c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 -- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ab806ecf0 msgr2=0x7f8ab80ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ab806ecf0 0x7f8ab80ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 -- 172.21.15.89:0/3399479139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac00b7f0 con 0x7f8ab80cc540 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab659c700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ab806ecf0 0x7f8ab80ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.156+0000 7f8ab5d9b700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab80cc540 0x7f8ab814acc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800c4e0 tx=0x7f8aa800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.157+0000 7f8aa6ffd700 1 -- 172.21.15.89:0/3399479139 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa80052d0 con 0x7f8ab80cc540 2023-11-24T06:18:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.157+0000 7f8aa6ffd700 1 -- 172.21.15.89:0/3399479139 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa8005430 con 0x7f8ab80cc540 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.157+0000 7f8aa6ffd700 1 -- 172.21.15.89:0/3399479139 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa80038c0 con 0x7f8ab80cc540 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.157+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab8150490 con 0x7f8ab80cc540 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.157+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab8150a50 con 0x7f8ab80cc540 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.158+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab8068d90 con 0x7f8ab80cc540 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.158+0000 7f8aa6ffd700 1 -- 172.21.15.89:0/3399479139 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8aa8003a20 con 0x7f8ab80cc540 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.158+0000 7f8aa6ffd700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8a9806aed0 0x7f8a9806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.158+0000 7f8aa6ffd700 1 -- 172.21.15.89:0/3399479139 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8aa80886e0 con 0x7f8ab80cc540 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.158+0000 7f8ab659c700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8a9806aed0 0x7f8a9806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.159+0000 7f8ab659c700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8a9806aed0 0x7f8a9806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00bee0 tx=0x7f8aac013000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.163+0000 7f8aa6ffd700 1 -- 172.21.15.89:0/3399479139 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8aa8053ae0 con 0x7f8ab80cc540 2023-11-24T06:18:46.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.353+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ab8068fb0 con 0x7f8ab80cc540 2023-11-24T06:18:46.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.353+0000 7f8aa6ffd700 1 -- 172.21.15.89:0/3399479139 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8aa8017020 con 0x7f8ab80cc540 2023-11-24T06:18:46.355 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:18:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8a9806aed0 msgr2=0x7f8a9806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8a9806aed0 0x7f8a9806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00bee0 tx=0x7f8aac013000 comp rx=0 tx=0).stop 2023-11-24T06:18:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab80cc540 msgr2=0x7f8ab814acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab80cc540 0x7f8ab814acc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800c4e0 tx=0x7f8aa800c7f0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 shutdown_connections 2023-11-24T06:18:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ab815a620 0x7f8ab81543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8a9806aed0 0x7f8a9806d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ab806ecf0 0x7f8ab80ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 --2- 172.21.15.89:0/3399479139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ab80cc540 0x7f8ab814acc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.355+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 >> 172.21.15.89:0/3399479139 conn(0x7f8ab80b2050 msgr2=0x7f8ab8155cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.356+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 shutdown_connections 2023-11-24T06:18:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.356+0000 7f8abcb92700 1 -- 172.21.15.89:0/3399479139 wait complete. 2023-11-24T06:18:46.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T06:18:46.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.487+0000 7fb23bb4b700 1 Processor -- start 2023-11-24T06:18:46.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.487+0000 7fb23bb4b700 1 -- start start 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23bb4b700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb2340adf00 0x7fb23406c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23bb4b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb23406c540 0x7fb2340b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23bb4b700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2340af950 con 0x7fb2341446a0 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23bb4b700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23406c9f0 con 0x7fb23406c540 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23bb4b700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb234144a80 con 0x7fb2340adf00 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb2398e7700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb2341446a0 0x7fb2340b5ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38396/0 (socket says 172.21.15.89:38396) 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23a0e8700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb23406c540 0x7fb2340b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb2398e7700 1 -- 172.21.15.89:0/320436261 learned_addr learned my addr 172.21.15.89:0/320436261 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb2390e6700 1 --2- 172.21.15.89:0/320436261 >> v2:172.21.15.123:3300/0 conn(0x7fb2340adf00 0x7fb23406c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23a0e8700 1 -- 172.21.15.89:0/320436261 >> v2:172.21.15.123:3300/0 conn(0x7fb2340adf00 msgr2=0x7fb23406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:46.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23a0e8700 1 --2- 172.21.15.89:0/320436261 >> v2:172.21.15.123:3300/0 conn(0x7fb2340adf00 0x7fb23406c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23a0e8700 1 -- 172.21.15.89:0/320436261 >> v1:172.21.15.89:6789/0 conn(0x7fb2341446a0 legacy=0x7fb2340b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb2388e5700 1 -- 172.21.15.89:0/320436261 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603745494 0 0) 0x7fb2340af950 con 0x7fb2341446a0 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.488+0000 7fb23a0e8700 1 -- 172.21.15.89:0/320436261 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340afc30 con 0x7fb23406c540 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23a0e8700 1 --2- 172.21.15.89:0/320436261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb2340b3d40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb23409f030 tx=0x7fb22c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84b86e26bddaf34 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb2388e5700 1 -- 172.21.15.89:0/320436261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00e070 con 0x7fb23406c540 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb2388e5700 1 -- 172.21.15.89:0/320436261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c00bae0 con 0x7fb23406c540 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb2388e5700 1 -- 172.21.15.89:0/320436261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00be90 con 0x7fb23406c540 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23bb4b700 1 -- 172.21.15.89:0/320436261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 msgr2=0x7fb2340b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/320436261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb2340b3d40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb23409f030 tx=0x7fb22c00b330 comp rx=0 tx=0).stop 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23bb4b700 1 -- 172.21.15.89:0/320436261 shutdown_connections 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/320436261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb2340b3d40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/320436261 >> v2:172.21.15.123:3300/0 conn(0x7fb2340adf00 0x7fb23406c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23bb4b700 1 -- 172.21.15.89:0/320436261 >> 172.21.15.89:0/320436261 conn(0x7fb234071b90 msgr2=0x7fb234071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:46.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23bb4b700 1 -- 172.21.15.89:0/320436261 shutdown_connections 2023-11-24T06:18:46.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.489+0000 7fb23bb4b700 1 -- 172.21.15.89:0/320436261 wait complete. 2023-11-24T06:18:46.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb23bb4b700 1 Processor -- start 2023-11-24T06:18:46.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb23bb4b700 1 -- start start 2023-11-24T06:18:46.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb23bb4b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb23415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb23bb4b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2340adf00 0x7fb234143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb23bb4b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb2341446a0 0x7fb234143950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb23bb4b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340cf3d0 con 0x7fb23406c540 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb23bb4b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340cf6b0 con 0x7fb2340adf00 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb23bb4b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340cf990 con 0x7fb2341446a0 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb2398e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb23415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb2398e7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb23415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34898/0 (socket says 172.21.15.89:34898) 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb2398e7700 1 -- 172.21.15.89:0/1411760582 learned_addr learned my addr 172.21.15.89:0/1411760582 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.491+0000 7fb2390e6700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2340adf00 0x7fb234143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb23a0e8700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb2341446a0 0x7fb234143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb2398e7700 1 -- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb2341446a0 msgr2=0x7fb234143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb2398e7700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb2341446a0 0x7fb234143950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb2398e7700 1 -- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2340adf00 msgr2=0x7fb234143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb2398e7700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2340adf00 0x7fb234143410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb2398e7700 1 -- 172.21.15.89:0/1411760582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c009a70 con 0x7fb23406c540 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb2390e6700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2340adf00 0x7fb234143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb2398e7700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb23415a2e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c550 tx=0x7fb22400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb22a7fc700 1 -- 172.21.15.89:0/1411760582 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224004c80 con 0x7fb23406c540 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb22a7fc700 1 -- 172.21.15.89:0/1411760582 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224004e00 con 0x7fb23406c540 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb22a7fc700 1 -- 172.21.15.89:0/1411760582 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224007310 con 0x7fb23406c540 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2340cfc70 con 0x7fb23406c540 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.492+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23415a820 con 0x7fb23406c540 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.493+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2340d02c0 con 0x7fb23406c540 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.493+0000 7fb22a7fc700 1 -- 172.21.15.89:0/1411760582 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb224007470 con 0x7fb23406c540 2023-11-24T06:18:46.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.494+0000 7fb22a7fc700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb21806aed0 0x7fb21806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.494+0000 7fb2390e6700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb21806aed0 0x7fb21806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.494+0000 7fb22a7fc700 1 -- 172.21.15.89:0/1411760582 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb224088f40 con 0x7fb23406c540 2023-11-24T06:18:46.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.494+0000 7fb2390e6700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb21806aed0 0x7fb21806d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb23415ab10 tx=0x7fb230008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.498+0000 7fb22a7fc700 1 -- 172.21.15.89:0/1411760582 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2240543d0 con 0x7fb23406c540 2023-11-24T06:18:46.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.690+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb2340753f0 con 0x7fb23406c540 2023-11-24T06:18:46.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.691+0000 7fb22a7fc700 1 -- 172.21.15.89:0/1411760582 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb2240579f0 con 0x7fb23406c540 2023-11-24T06:18:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:46.693 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-11-24T06:18:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:18:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2023-11-24T06:18:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2023-11-24T06:18:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:18:46.693 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-11-24T06:18:46.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb21806aed0 msgr2=0x7fb21806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb21806aed0 0x7fb21806d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb23415ab10 tx=0x7fb230008040 comp rx=0 tx=0).stop 2023-11-24T06:18:46.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 msgr2=0x7fb23415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb23415a2e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c550 tx=0x7fb22400c910 comp rx=0 tx=0).stop 2023-11-24T06:18:46.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 shutdown_connections 2023-11-24T06:18:46.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb2341446a0 0x7fb234143950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23406c540 0x7fb23415a2e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb21806aed0 0x7fb21806d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 --2- 172.21.15.89:0/1411760582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2340adf00 0x7fb234143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 >> 172.21.15.89:0/1411760582 conn(0x7fb234071b90 msgr2=0x7fb234073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:46.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 shutdown_connections 2023-11-24T06:18:46.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.693+0000 7fb23bb4b700 1 -- 172.21.15.89:0/1411760582 wait complete. 2023-11-24T06:18:46.706 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T06:18:46.707 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T06:18:46.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.829+0000 7f2f062d9700 1 Processor -- start 2023-11-24T06:18:46.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.829+0000 7f2f062d9700 1 -- start start 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.829+0000 7f2f062d9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2f000b03b0 0x7f2f000cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.829+0000 7f2f062d9700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f00161c80 con 0x7f2f0015e020 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.829+0000 7f2f062d9700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f00161f40 con 0x7f2f000b18f0 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.829+0000 7f2f062d9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f000d0020 con 0x7f2f000b03b0 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efffff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2f0015e020 0x7f2f000b11e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38410/0 (socket says 172.21.15.89:38410) 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efffff700 1 -- 172.21.15.89:0/2766167406 learned_addr learned my addr 172.21.15.89:0/2766167406 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177882723 0 0) 0x7f2f00161c80 con 0x7f2f0015e020 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee80035b0 con 0x7f2f0015e020 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2f04876700 1 --2- 172.21.15.89:0/2766167406 >> v2:172.21.15.123:3300/0 conn(0x7f2f000b03b0 0x7f2f000cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2412220890 0 0) 0x7f2ee80035b0 con 0x7f2f0015e020 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 >> v2:172.21.15.123:3300/0 conn(0x7f2f000b03b0 msgr2=0x7f2f000cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 --2- 172.21.15.89:0/2766167406 >> v2:172.21.15.123:3300/0 conn(0x7f2f000b03b0 0x7f2f000cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 >> v1:172.21.15.123:6789/0 conn(0x7f2f000b18f0 legacy=0x7f2f000cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00162220 con 0x7f2f0015e020 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f2ef0001670 con 0x7f2f0015e020 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 >> v1:172.21.15.89:6789/0 conn(0x7f2f0015e020 legacy=0x7f2f000b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8003a00 0x7f2ee8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee8004850 0x7f2ee8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000d0020 con 0x7f2ee8005210 2023-11-24T06:18:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee800c1b0 con 0x7f2ee8004850 2023-11-24T06:18:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee800c370 con 0x7f2ee8003a00 2023-11-24T06:18:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2f04876700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2efffff700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8003a00 0x7f2ee8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.830+0000 7f2eff7fe700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee8004850 0x7f2ee8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f04876700 1 -- 172.21.15.89:0/2766167406 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8003a00 msgr2=0x7f2ee8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f04876700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8003a00 0x7f2ee8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f04876700 1 -- 172.21.15.89:0/2766167406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee8004850 msgr2=0x7f2ee8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f04876700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee8004850 0x7f2ee8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f04876700 1 -- 172.21.15.89:0/2766167406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80035b0 con 0x7f2ee8005210 2023-11-24T06:18:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f04876700 1 -- 172.21.15.89:0/2766167406 reap_dead start 2023-11-24T06:18:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f04876700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009f90 tx=0x7f2ee8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac2da6fb21c3a8b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8003a20 con 0x7f2ee8005210 2023-11-24T06:18:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef800f040 con 0x7f2ee8005210 2023-11-24T06:18:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2efeffd700 1 -- 172.21.15.89:0/2766167406 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8003de0 con 0x7f2ee8005210 2023-11-24T06:18:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f062d9700 1 -- 172.21.15.89:0/2766167406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee8005210 msgr2=0x7f2ee800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f062d9700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009f90 tx=0x7f2ee8005880 comp rx=0 tx=0).stop 2023-11-24T06:18:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f062d9700 1 -- 172.21.15.89:0/2766167406 shutdown_connections 2023-11-24T06:18:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f062d9700 1 --2- 172.21.15.89:0/2766167406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f062d9700 1 -- 172.21.15.89:0/2766167406 >> 172.21.15.89:0/2766167406 conn(0x7f2f00072710 msgr2=0x7f2f00072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f062d9700 1 -- 172.21.15.89:0/2766167406 shutdown_connections 2023-11-24T06:18:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.831+0000 7f2f062d9700 1 -- 172.21.15.89:0/2766167406 wait complete. 2023-11-24T06:18:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.832+0000 7f2f062d9700 1 Processor -- start 2023-11-24T06:18:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.833+0000 7f2f062d9700 1 -- start start 2023-11-24T06:18:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.833+0000 7f2f062d9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8005210 0x7f2f001f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.833+0000 7f2f062d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f0014fdc0 0x7f2f001f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.833+0000 7f2f062d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0015bda0 0x7f2f0015c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.833+0000 7f2f062d9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0015c770 con 0x7f2f0015bda0 2023-11-24T06:18:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.833+0000 7f2f062d9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001f3720 con 0x7f2f0014fdc0 2023-11-24T06:18:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.833+0000 7f2f062d9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001f7d00 con 0x7f2ee8005210 2023-11-24T06:18:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0015bda0 0x7f2f0015c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0015bda0 0x7f2f0015c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34922/0 (socket says 172.21.15.89:34922) 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 -- 172.21.15.89:0/4118020789 learned_addr learned my addr 172.21.15.89:0/4118020789 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2eff7fe700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f0014fdc0 0x7f2f001f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2efffff700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8005210 0x7f2f001f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 -- 172.21.15.89:0/4118020789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8005210 msgr2=0x7f2f001f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8005210 0x7f2f001f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 -- 172.21.15.89:0/4118020789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f0014fdc0 msgr2=0x7f2f001f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f0014fdc0 0x7f2f001f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 -- 172.21.15.89:0/4118020789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f001f7740 con 0x7f2f0015bda0 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f04876700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0015bda0 0x7f2f0015c230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800e700 tx=0x7f2ef8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2efcff9700 1 -- 172.21.15.89:0/4118020789 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8003e10 con 0x7f2f0015bda0 2023-11-24T06:18:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2efcff9700 1 -- 172.21.15.89:0/4118020789 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef8004230 con 0x7f2f0015bda0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f001f7a20 con 0x7f2f0015bda0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2efcff9700 1 -- 172.21.15.89:0/4118020789 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef80045e0 con 0x7f2f0015bda0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.834+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f000c40c0 con 0x7f2f0015bda0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.835+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f000c4280 con 0x7f2f0015bda0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.836+0000 7f2efcff9700 1 -- 172.21.15.89:0/4118020789 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2ef8004740 con 0x7f2f0015bda0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.836+0000 7f2efcff9700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2eec06aed0 0x7f2eec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.836+0000 7f2efffff700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2eec06aed0 0x7f2eec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.836+0000 7f2efcff9700 1 -- 172.21.15.89:0/4118020789 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ef808a440 con 0x7f2f0015bda0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.837+0000 7f2efffff700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2eec06aed0 0x7f2eec06d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0000c00 tx=0x7f2ef0000ed0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:46.841+0000 7f2efcff9700 1 -- 172.21.15.89:0/4118020789 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ef80559f0 con 0x7f2f0015bda0 2023-11-24T06:18:47.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.030+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2f00067f60 con 0x7f2f0015bda0 2023-11-24T06:18:47.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:18:47.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:18:47.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.987+0000 7f2efcff9700 1 -- 172.21.15.89:0/4118020789 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2ef8059010 con 0x7f2f0015bda0 2023-11-24T06:18:47.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2eec06aed0 msgr2=0x7f2eec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:47.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2eec06aed0 0x7f2eec06d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0000c00 tx=0x7f2ef0000ed0 comp rx=0 tx=0).stop 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0015bda0 msgr2=0x7f2f0015c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0015bda0 0x7f2f0015c230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800e700 tx=0x7f2ef8004b40 comp rx=0 tx=0).stop 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 shutdown_connections 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0015bda0 0x7f2f0015c230 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee8005210 0x7f2f001f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2eec06aed0 0x7f2eec06d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.990+0000 7f2f062d9700 1 --2- 172.21.15.89:0/4118020789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f0014fdc0 0x7f2f001f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.991+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 >> 172.21.15.89:0/4118020789 conn(0x7f2f00072710 msgr2=0x7f2f000aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:47.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.991+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 shutdown_connections 2023-11-24T06:18:47.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:47.991+0000 7f2f062d9700 1 -- 172.21.15.89:0/4118020789 wait complete. 2023-11-24T06:18:47.994 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2023-11-24T06:18:48.010 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T06:18:48.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.141+0000 7fe02a651700 1 Processor -- start 2023-11-24T06:18:48.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.141+0000 7fe02a651700 1 -- start start 2023-11-24T06:18:48.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe02a651700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe02415b370 0x7fe02415b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:48.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe02a651700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0240ae5e0 con 0x7fe0240b3c60 2023-11-24T06:18:48.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe02a651700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02415be30 con 0x7fe0241587f0 2023-11-24T06:18:48.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe02a651700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02415c110 con 0x7fe02415b370 2023-11-24T06:18:48.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe028bee700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe0240b3c60 0x7fe0240af060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38434/0 (socket says 172.21.15.89:38434) 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe028bee700 1 -- 172.21.15.89:0/1302115251 learned_addr learned my addr 172.21.15.89:0/1302115251 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe0237fe700 1 --2- 172.21.15.89:0/1302115251 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe02415b370 0x7fe02415b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe022ffd700 1 -- 172.21.15.89:0/1302115251 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310534932 0 0) 0x7fe0240ae5e0 con 0x7fe0240b3c60 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.142+0000 7fe022ffd700 1 -- 172.21.15.89:0/1302115251 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe00c0035b0 con 0x7fe0240b3c60 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe022ffd700 1 -- 172.21.15.89:0/1302115251 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383181128 0 0) 0x7fe02415be30 con 0x7fe0241587f0 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe022ffd700 1 -- 172.21.15.89:0/1302115251 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0240ae5e0 con 0x7fe0241587f0 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe022ffd700 1 -- 172.21.15.89:0/1302115251 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 416383093 0 0) 0x7fe00c0035b0 con 0x7fe0240b3c60 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe0237fe700 1 -- 172.21.15.89:0/1302115251 >> v1:172.21.15.123:6789/0 conn(0x7fe0241587f0 legacy=0x7fe02406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe0237fe700 1 -- 172.21.15.89:0/1302115251 >> v1:172.21.15.89:6789/0 conn(0x7fe0240b3c60 legacy=0x7fe0240af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe0237fe700 1 -- 172.21.15.89:0/1302115251 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0240ae860 con 0x7fe02415b370 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe0237fe700 1 --2- 172.21.15.89:0/1302115251 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe02415b370 0x7fe02415b7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe014009870 tx=0x7fe014009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6fe8aa48688b7f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe022ffd700 1 -- 172.21.15.89:0/1302115251 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c0019f0 con 0x7fe02415b370 2023-11-24T06:18:48.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe022ffd700 1 -- 172.21.15.89:0/1302115251 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe014004030 con 0x7fe02415b370 2023-11-24T06:18:48.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.143+0000 7fe022ffd700 1 -- 172.21.15.89:0/1302115251 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0140043a0 con 0x7fe02415b370 2023-11-24T06:18:48.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.144+0000 7fe02a651700 1 -- 172.21.15.89:0/1302115251 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe02415b370 msgr2=0x7fe02415b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:48.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.144+0000 7fe02a651700 1 --2- 172.21.15.89:0/1302115251 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe02415b370 0x7fe02415b7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe014009870 tx=0x7fe014009b80 comp rx=0 tx=0).stop 2023-11-24T06:18:48.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.144+0000 7fe02a651700 1 -- 172.21.15.89:0/1302115251 shutdown_connections 2023-11-24T06:18:48.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.144+0000 7fe02a651700 1 --2- 172.21.15.89:0/1302115251 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe02415b370 0x7fe02415b7d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:48.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.144+0000 7fe02a651700 1 -- 172.21.15.89:0/1302115251 >> 172.21.15.89:0/1302115251 conn(0x7fe0240d0bf0 msgr2=0x7fe0240d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:48.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.144+0000 7fe02a651700 1 -- 172.21.15.89:0/1302115251 shutdown_connections 2023-11-24T06:18:48.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.144+0000 7fe02a651700 1 -- 172.21.15.89:0/1302115251 wait complete. 2023-11-24T06:18:48.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.145+0000 7fe02a651700 1 Processor -- start 2023-11-24T06:18:48.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.145+0000 7fe02a651700 1 -- start start 2023-11-24T06:18:48.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe02a651700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0240b3c60 0x7fe0241e6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:48.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe02a651700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0241587f0 0x7fe02415c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:48.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe02a651700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02415b370 0x7fe02415a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:48.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe02a651700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0241e7450 con 0x7fe0241587f0 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe02a651700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02415cba0 con 0x7fe02415b370 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe02a651700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02415ce80 con 0x7fe0240b3c60 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0241587f0 0x7fe02415c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0241587f0 0x7fe02415c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34934/0 (socket says 172.21.15.89:34934) 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 -- 172.21.15.89:0/4222970038 learned_addr learned my addr 172.21.15.89:0/4222970038 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe028bee700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02415b370 0x7fe02415a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 -- 172.21.15.89:0/4222970038 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0240b3c60 msgr2=0x7fe0241e6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0240b3c60 0x7fe0241e6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 -- 172.21.15.89:0/4222970038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02415b370 msgr2=0x7fe02415a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02415b370 0x7fe02415a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 -- 172.21.15.89:0/4222970038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0140094a0 con 0x7fe0241587f0 2023-11-24T06:18:48.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe028bee700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02415b370 0x7fe02415a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe0237fe700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0241587f0 0x7fe02415c350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0240ae790 tx=0x7fe014004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe020ff9700 1 -- 172.21.15.89:0/4222970038 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe014003bb0 con 0x7fe0241587f0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe020ff9700 1 -- 172.21.15.89:0/4222970038 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01401f070 con 0x7fe0241587f0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.146+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02415d160 con 0x7fe0241587f0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.147+0000 7fe020ff9700 1 -- 172.21.15.89:0/4222970038 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe014011630 con 0x7fe0241587f0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.147+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0241e38e0 con 0x7fe0241587f0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.148+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0241e3a70 con 0x7fe0241587f0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.148+0000 7fe020ff9700 1 -- 172.21.15.89:0/4222970038 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe014021030 con 0x7fe0241587f0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.148+0000 7fe020ff9700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe01006aed0 0x7fe01006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.148+0000 7fe020ff9700 1 -- 172.21.15.89:0/4222970038 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe01408a7f0 con 0x7fe0241587f0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.148+0000 7fe023fff700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe01006aed0 0x7fe01006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:48.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.149+0000 7fe023fff700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe01006aed0 0x7fe01006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe0180010e0 tx=0x7fe0180079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:48.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.153+0000 7fe020ff9700 1 -- 172.21.15.89:0/4222970038 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe014055c30 con 0x7fe0241587f0 2023-11-24T06:18:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.343+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe024068fb0 con 0x7fe0241587f0 2023-11-24T06:18:48.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.990+0000 7fe020ff9700 1 -- 172.21.15.89:0/4222970038 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe014059250 con 0x7fe0241587f0 2023-11-24T06:18:48.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe01006aed0 msgr2=0x7fe01006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:48.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe01006aed0 0x7fe01006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe0180010e0 tx=0x7fe0180079e0 comp rx=0 tx=0).stop 2023-11-24T06:18:48.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0241587f0 msgr2=0x7fe02415c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0241587f0 0x7fe02415c350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0240ae790 tx=0x7fe014004750 comp rx=0 tx=0).stop 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 shutdown_connections 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe01006aed0 0x7fe01006d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02415b370 0x7fe02415a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0240b3c60 0x7fe0241e6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 --2- 172.21.15.89:0/4222970038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0241587f0 0x7fe02415c350 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 >> 172.21.15.89:0/4222970038 conn(0x7fe0240d0bf0 msgr2=0x7fe0240b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 shutdown_connections 2023-11-24T06:18:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:48.993+0000 7fe02a651700 1 -- 172.21.15.89:0/4222970038 wait complete. 2023-11-24T06:18:49.002 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2023-11-24T06:18:49.021 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T06:18:49.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.146+0000 7f1b8a202700 1 Processor -- start 2023-11-24T06:18:49.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.146+0000 7f1b8a202700 1 -- start start 2023-11-24T06:18:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b8a202700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b8415e080 0x7f1b8406bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b8a202700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1b840ce430 0x7f1b840ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b8a202700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8406ba50 con 0x7f1b840b4310 2023-11-24T06:18:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b8a202700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840d1200 con 0x7f1b840ce430 2023-11-24T06:18:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b8a202700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840cee20 con 0x7f1b8415e080 2023-11-24T06:18:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b837fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b8415e080 0x7f1b8406bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b837fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b8415e080 0x7f1b8406bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:43816/0 (socket says 172.21.15.89:43816) 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b82ffd700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f1b840b4310 0x7f1b840d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:42268/0 (socket says 172.21.15.89:42268) 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b83fff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1b840ce430 0x7f1b840ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.147+0000 7f1b837fe700 1 -- 172.21.15.89:0/1168740467 learned_addr learned my addr 172.21.15.89:0/1168740467 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b827fc700 1 -- 172.21.15.89:0/1168740467 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293597608 0 0) 0x7f1b8406ba50 con 0x7f1b840b4310 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b827fc700 1 -- 172.21.15.89:0/1168740467 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c0035b0 con 0x7f1b840b4310 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b83fff700 1 -- 172.21.15.89:0/1168740467 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b8415e080 msgr2=0x7f1b8406bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b83fff700 1 --2- 172.21.15.89:0/1168740467 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b8415e080 0x7f1b8406bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b83fff700 1 -- 172.21.15.89:0/1168740467 >> v1:172.21.15.123:6789/0 conn(0x7f1b840b4310 legacy=0x7f1b840d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b83fff700 1 -- 172.21.15.89:0/1168740467 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8406b120 con 0x7f1b840ce430 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b837fe700 1 --2- 172.21.15.89:0/1168740467 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b8415e080 0x7f1b8406bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b83fff700 1 --2- 172.21.15.89:0/1168740467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b840ce430 0x7f1b840ce8e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009d90 tx=0x7f1b7c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a2aaf0ea7ed0ce5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b827fc700 1 -- 172.21.15.89:0/1168740467 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c00e070 con 0x7f1b840ce430 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b827fc700 1 -- 172.21.15.89:0/1168740467 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c00bb20 con 0x7f1b840ce430 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b827fc700 1 -- 172.21.15.89:0/1168740467 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c012430 con 0x7f1b840ce430 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.148+0000 7f1b8a202700 1 -- 172.21.15.89:0/1168740467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b840ce430 msgr2=0x7f1b840ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.149+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1168740467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b840ce430 0x7f1b840ce8e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009d90 tx=0x7f1b7c00b330 comp rx=0 tx=0).stop 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.149+0000 7f1b8a202700 1 -- 172.21.15.89:0/1168740467 shutdown_connections 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.149+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1168740467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b840ce430 0x7f1b840ce8e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.149+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1168740467 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b8415e080 0x7f1b8406bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.149+0000 7f1b8a202700 1 -- 172.21.15.89:0/1168740467 >> 172.21.15.89:0/1168740467 conn(0x7f1b840b3780 msgr2=0x7f1b840b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.149+0000 7f1b8a202700 1 -- 172.21.15.89:0/1168740467 shutdown_connections 2023-11-24T06:18:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.149+0000 7f1b8a202700 1 -- 172.21.15.89:0/1168740467 wait complete. 2023-11-24T06:18:49.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.150+0000 7f1b8a202700 1 Processor -- start 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.150+0000 7f1b8a202700 1 -- start start 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b8a202700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b840b4310 0x7f1b840c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b8a202700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b840ce430 0x7f1b840c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b8a202700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b8415e080 0x7f1b840c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b8a202700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840c7ba0 con 0x7f1b840b4310 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b8a202700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840c9700 con 0x7f1b8415e080 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b8a202700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8415ff10 con 0x7f1b840ce430 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b840b4310 0x7f1b840c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b840b4310 0x7f1b840c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34946/0 (socket says 172.21.15.89:34946) 2023-11-24T06:18:49.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 -- 172.21.15.89:0/1232345048 learned_addr learned my addr 172.21.15.89:0/1232345048 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b82ffd700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b840ce430 0x7f1b840c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b83fff700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b8415e080 0x7f1b840c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 -- 172.21.15.89:0/1232345048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b840ce430 msgr2=0x7f1b840c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b840ce430 0x7f1b840c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 -- 172.21.15.89:0/1232345048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b8415e080 msgr2=0x7f1b840c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b8415e080 0x7f1b840c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 -- 172.21.15.89:0/1232345048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b78009550 con 0x7f1b840b4310 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b837fe700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b840b4310 0x7f1b840c7fc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800e9b0 tx=0x7f1b7800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b6bfff700 1 -- 172.21.15.89:0/1232345048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7800ee50 con 0x7f1b840b4310 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b6bfff700 1 -- 172.21.15.89:0/1232345048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7800c850 con 0x7f1b840b4310 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7c009a70 con 0x7f1b840b4310 2023-11-24T06:18:49.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b6bfff700 1 -- 172.21.15.89:0/1232345048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7800cbc0 con 0x7f1b840b4310 2023-11-24T06:18:49.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.151+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b840c7200 con 0x7f1b840b4310 2023-11-24T06:18:49.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.152+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b840c55d0 con 0x7f1b840b4310 2023-11-24T06:18:49.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.153+0000 7f1b6bfff700 1 -- 172.21.15.89:0/1232345048 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1b7800cd20 con 0x7f1b840b4310 2023-11-24T06:18:49.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.153+0000 7f1b6bfff700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b7006aed0 0x7f1b7006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:49.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.154+0000 7f1b82ffd700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b7006aed0 0x7f1b7006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:49.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.154+0000 7f1b6bfff700 1 -- 172.21.15.89:0/1232345048 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1b780894f0 con 0x7f1b840b4310 2023-11-24T06:18:49.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.154+0000 7f1b82ffd700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b7006aed0 0x7f1b7006d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1b74007fd0 tx=0x7f1b74007dc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:49.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.158+0000 7f1b6bfff700 1 -- 172.21.15.89:0/1232345048 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b78054690 con 0x7f1b840b4310 2023-11-24T06:18:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:49.390+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1b840c8af0 con 0x7f1b840b4310 2023-11-24T06:18:50.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.014+0000 7f1b6bfff700 1 -- 172.21.15.89:0/1232345048 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1b78057cb0 con 0x7f1b840b4310 2023-11-24T06:18:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.028+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b7006aed0 msgr2=0x7f1b7006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.028+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b7006aed0 0x7f1b7006d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1b74007fd0 tx=0x7f1b74007dc0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.028+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b840b4310 msgr2=0x7f1b840c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.028+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b840b4310 0x7f1b840c7fc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800e9b0 tx=0x7f1b7800ecc0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.029+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 shutdown_connections 2023-11-24T06:18:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.029+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b7006aed0 0x7f1b7006d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.029+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b8415e080 0x7f1b840c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.029+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b840b4310 0x7f1b840c7fc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.029+0000 7f1b8a202700 1 --2- 172.21.15.89:0/1232345048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b840ce430 0x7f1b840c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.029+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 >> 172.21.15.89:0/1232345048 conn(0x7f1b840b3780 msgr2=0x7f1b840cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.029+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 shutdown_connections 2023-11-24T06:18:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.031+0000 7f1b8a202700 1 -- 172.21.15.89:0/1232345048 wait complete. 2023-11-24T06:18:50.036 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 2 and data pool 3 2023-11-24T06:18:50.051 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:50.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.171+0000 7f573c7a4700 1 Processor -- start 2023-11-24T06:18:50.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.171+0000 7f573c7a4700 1 -- start start 2023-11-24T06:18:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573c7a4700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57340b0ba0 0x7f57340af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573c7a4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57340b0f80 0x7f57340cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573c7a4700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57340b2bd0 con 0x7f573415ee90 2023-11-24T06:18:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573c7a4700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57340b1fc0 con 0x7f57340b0ba0 2023-11-24T06:18:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573c7a4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57340b22a0 con 0x7f57340b0f80 2023-11-24T06:18:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573a540700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57340b0ba0 0x7f57340af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573a540700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57340b0ba0 0x7f57340af5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34954/0 (socket says 172.21.15.89:34954) 2023-11-24T06:18:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573a540700 1 -- 172.21.15.89:0/787951257 learned_addr learned my addr 172.21.15.89:0/787951257 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f573ad41700 1 --1- 172.21.15.89:0/787951257 >> v1:172.21.15.89:6789/0 conn(0x7f573415ee90 0x7f57340ccca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38462/0 (socket says 172.21.15.89:38462) 2023-11-24T06:18:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.172+0000 7f5739d3f700 1 --2- 172.21.15.89:0/787951257 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57340b0f80 0x7f57340cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573a540700 1 -- 172.21.15.89:0/787951257 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57340b0f80 msgr2=0x7f57340cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573a540700 1 --2- 172.21.15.89:0/787951257 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57340b0f80 0x7f57340cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573a540700 1 -- 172.21.15.89:0/787951257 >> v1:172.21.15.89:6789/0 conn(0x7f573415ee90 legacy=0x7f57340ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573953e700 1 -- 172.21.15.89:0/787951257 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021734642 0 0) 0x7f57340b2bd0 con 0x7f573415ee90 2023-11-24T06:18:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573a540700 1 -- 172.21.15.89:0/787951257 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57340b2580 con 0x7f57340b0ba0 2023-11-24T06:18:50.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f5739d3f700 1 --2- 172.21.15.89:0/787951257 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57340b0f80 0x7f57340cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:50.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573a540700 1 --2- 172.21.15.89:0/787951257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0ba0 0x7f57340af5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f57240098f0 tx=0x7f5724009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27558a5a24b1495 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573953e700 1 -- 172.21.15.89:0/787951257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572400b8a0 con 0x7f57340b0ba0 2023-11-24T06:18:50.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573953e700 1 -- 172.21.15.89:0/787951257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572400ba00 con 0x7f57340b0ba0 2023-11-24T06:18:50.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573953e700 1 -- 172.21.15.89:0/787951257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572400bd90 con 0x7f57340b0ba0 2023-11-24T06:18:50.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573c7a4700 1 -- 172.21.15.89:0/787951257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0ba0 msgr2=0x7f57340af5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573c7a4700 1 --2- 172.21.15.89:0/787951257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0ba0 0x7f57340af5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f57240098f0 tx=0x7f5724009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:50.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573c7a4700 1 -- 172.21.15.89:0/787951257 shutdown_connections 2023-11-24T06:18:50.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573c7a4700 1 --2- 172.21.15.89:0/787951257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0ba0 0x7f57340af5c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573c7a4700 1 --2- 172.21.15.89:0/787951257 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57340b0f80 0x7f57340cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573c7a4700 1 -- 172.21.15.89:0/787951257 >> 172.21.15.89:0/787951257 conn(0x7f57340b57b0 msgr2=0x7f57340b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:50.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573c7a4700 1 -- 172.21.15.89:0/787951257 shutdown_connections 2023-11-24T06:18:50.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.173+0000 7f573c7a4700 1 -- 172.21.15.89:0/787951257 wait complete. 2023-11-24T06:18:50.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f573c7a4700 1 Processor -- start 2023-11-24T06:18:50.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f573c7a4700 1 -- start start 2023-11-24T06:18:50.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f573c7a4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57340b0ba0 0x7f573409c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f573c7a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0f80 0x7f573409a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f573c7a4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f573415ee90 0x7f573409a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f573c7a4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573409c790 con 0x7f57340b0f80 2023-11-24T06:18:50.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f573c7a4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734153c50 con 0x7f57340b0ba0 2023-11-24T06:18:50.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f573c7a4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573409c9e0 con 0x7f573415ee90 2023-11-24T06:18:50.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f5739d3f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0f80 0x7f573409a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f5739d3f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0f80 0x7f573409a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34962/0 (socket says 172.21.15.89:34962) 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.175+0000 7f5739d3f700 1 -- 172.21.15.89:0/4001649396 learned_addr learned my addr 172.21.15.89:0/4001649396 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f573ad41700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f573415ee90 0x7f573409a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f573a540700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57340b0ba0 0x7f573409c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f5739d3f700 1 -- 172.21.15.89:0/4001649396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f573415ee90 msgr2=0x7f573409a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f5739d3f700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f573415ee90 0x7f573409a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f5739d3f700 1 -- 172.21.15.89:0/4001649396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57340b0ba0 msgr2=0x7f573409c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f5739d3f700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57340b0ba0 0x7f573409c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f5739d3f700 1 -- 172.21.15.89:0/4001649396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5724009550 con 0x7f57340b0f80 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f5739d3f700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0f80 0x7f573409a380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f573000c750 tx=0x7f573000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f572affd700 1 -- 172.21.15.89:0/4001649396 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730011800 con 0x7f57340b0f80 2023-11-24T06:18:50.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f572affd700 1 -- 172.21.15.89:0/4001649396 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5730011960 con 0x7f57340b0f80 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573409ccc0 con 0x7f57340b0f80 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f572affd700 1 -- 172.21.15.89:0/4001649396 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730011cd0 con 0x7f57340b0f80 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.176+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573409d280 con 0x7f57340b0f80 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.177+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5734067f60 con 0x7f57340b0f80 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.177+0000 7f572affd700 1 -- 172.21.15.89:0/4001649396 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5730021350 con 0x7f57340b0f80 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.178+0000 7f572affd700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f571806aed0 0x7f571806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.178+0000 7f572affd700 1 -- 172.21.15.89:0/4001649396 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5730089380 con 0x7f57340b0f80 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.178+0000 7f573a540700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f571806aed0 0x7f571806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.178+0000 7f573a540700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f571806aed0 0x7f571806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5724009c40 tx=0x7f5724008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.182+0000 7f572affd700 1 -- 172.21.15.89:0/4001649396 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57300544c0 con 0x7f57340b0f80 2023-11-24T06:18:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.377+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5734071f00 con 0x7f57340b0f80 2023-11-24T06:18:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.378+0000 7f572affd700 1 -- 172.21.15.89:0/4001649396 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16482 (secure 0 0 0) 0x7f5730057ae0 con 0x7f57340b0f80 2023-11-24T06:18:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:50.380 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:49.390863+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f571806aed0 msgr2=0x7f571806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f571806aed0 0x7f571806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5724009c40 tx=0x7f5724008030 comp rx=0 tx=0).stop 2023-11-24T06:18:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0f80 msgr2=0x7f573409a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0f80 0x7f573409a380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f573000c750 tx=0x7f573000ca60 comp rx=0 tx=0).stop 2023-11-24T06:18:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 shutdown_connections 2023-11-24T06:18:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f573415ee90 0x7f573409a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f571806aed0 0x7f571806d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57340b0ba0 0x7f573409c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 --2- 172.21.15.89:0/4001649396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57340b0f80 0x7f573409a380 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 >> 172.21.15.89:0/4001649396 conn(0x7f57340b57b0 msgr2=0x7f57340cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 shutdown_connections 2023-11-24T06:18:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.380+0000 7f573c7a4700 1 -- 172.21.15.89:0/4001649396 wait complete. 2023-11-24T06:18:50.394 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:50.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.511+0000 7ff833781700 1 Processor -- start 2023-11-24T06:18:50.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.511+0000 7ff833781700 1 -- start start 2023-11-24T06:18:50.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.512+0000 7ff833781700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 0x7ff82c1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.512+0000 7ff833781700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff82c0b3370 0x7ff82c0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.512+0000 7ff833781700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c0b2030 con 0x7ff82c0d1c70 2023-11-24T06:18:50.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.512+0000 7ff833781700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff82c0b2f50 con 0x7ff82c0b3370 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.512+0000 7ff833781700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c0b22d0 con 0x7ff82c15bdd0 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.512+0000 7ff830d1c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff82c0b3370 0x7ff82c0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.512+0000 7ff830d1c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff82c0b3370 0x7ff82c0b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34970/0 (socket says 172.21.15.89:34970) 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.512+0000 7ff830d1c700 1 -- 172.21.15.89:0/7591867 learned_addr learned my addr 172.21.15.89:0/7591867 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff83151d700 1 --2- 172.21.15.89:0/7591867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 0x7ff82c1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff830d1c700 1 -- 172.21.15.89:0/7591867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 msgr2=0x7ff82c1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff830d1c700 1 --2- 172.21.15.89:0/7591867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 0x7ff82c1604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff830d1c700 1 -- 172.21.15.89:0/7591867 >> v1:172.21.15.123:6789/0 conn(0x7ff82c0d1c70 legacy=0x7ff82c0ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff830d1c700 1 -- 172.21.15.89:0/7591867 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c0b25b0 con 0x7ff82c0b3370 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff830d1c700 1 --2- 172.21.15.89:0/7591867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c0b1af0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff8280098f0 tx=0x7ff828009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd510e1553a675cf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff823fff700 1 -- 172.21.15.89:0/7591867 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828004230 con 0x7ff82c0b3370 2023-11-24T06:18:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff823fff700 1 -- 172.21.15.89:0/7591867 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff828004390 con 0x7ff82c0b3370 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff823fff700 1 -- 172.21.15.89:0/7591867 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8280036a0 con 0x7ff82c0b3370 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff833781700 1 -- 172.21.15.89:0/7591867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 msgr2=0x7ff82c0b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.513+0000 7ff833781700 1 --2- 172.21.15.89:0/7591867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c0b1af0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff8280098f0 tx=0x7ff828009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.514+0000 7ff833781700 1 -- 172.21.15.89:0/7591867 shutdown_connections 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.514+0000 7ff833781700 1 --2- 172.21.15.89:0/7591867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 0x7ff82c1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.514+0000 7ff833781700 1 --2- 172.21.15.89:0/7591867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c0b1af0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.514+0000 7ff833781700 1 -- 172.21.15.89:0/7591867 >> 172.21.15.89:0/7591867 conn(0x7ff82c0731a0 msgr2=0x7ff82c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.514+0000 7ff833781700 1 -- 172.21.15.89:0/7591867 shutdown_connections 2023-11-24T06:18:50.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.514+0000 7ff833781700 1 -- 172.21.15.89:0/7591867 wait complete. 2023-11-24T06:18:50.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.515+0000 7ff833781700 1 Processor -- start 2023-11-24T06:18:50.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.515+0000 7ff833781700 1 -- start start 2023-11-24T06:18:50.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.515+0000 7ff833781700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff833781700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff82c0d1c70 0x7ff82c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff833781700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 0x7ff82c0cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff833781700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c0b7610 con 0x7ff82c0b3370 2023-11-24T06:18:50.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff833781700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c0cb740 con 0x7ff82c0d1c70 2023-11-24T06:18:50.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff833781700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c1576a0 con 0x7ff82c15bdd0 2023-11-24T06:18:50.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34974/0 (socket says 172.21.15.89:34974) 2023-11-24T06:18:50.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 -- 172.21.15.89:0/2861672787 learned_addr learned my addr 172.21.15.89:0/2861672787 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:50.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff830d1c700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff82c0d1c70 0x7ff82c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 -- 172.21.15.89:0/2861672787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 msgr2=0x7ff82c0cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 0x7ff82c0cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 -- 172.21.15.89:0/2861672787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff82c0d1c70 msgr2=0x7ff82c0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff82c0d1c70 0x7ff82c0cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 -- 172.21.15.89:0/2861672787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff828009550 con 0x7ff82c0b3370 2023-11-24T06:18:50.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff830d1c700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff82c0d1c70 0x7ff82c0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:50.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff83151d700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c157160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00c820 tx=0x7ff81c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff821ffb700 1 -- 172.21.15.89:0/2861672787 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c011800 con 0x7ff82c0b3370 2023-11-24T06:18:50.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff821ffb700 1 -- 172.21.15.89:0/2861672787 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81c011960 con 0x7ff82c0b3370 2023-11-24T06:18:50.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82c0b5980 con 0x7ff82c0b3370 2023-11-24T06:18:50.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff821ffb700 1 -- 172.21.15.89:0/2861672787 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c011cf0 con 0x7ff82c0b3370 2023-11-24T06:18:50.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.516+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c0b5f40 con 0x7ff82c0b3370 2023-11-24T06:18:50.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.517+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82c067f60 con 0x7ff82c0b3370 2023-11-24T06:18:50.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.518+0000 7ff821ffb700 1 -- 172.21.15.89:0/2861672787 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff81c018360 con 0x7ff82c0b3370 2023-11-24T06:18:50.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.518+0000 7ff821ffb700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff81006aed0 0x7ff81006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.518+0000 7ff821ffb700 1 -- 172.21.15.89:0/2861672787 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff81c089480 con 0x7ff82c0b3370 2023-11-24T06:18:50.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.518+0000 7ff830d1c700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff81006aed0 0x7ff81006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.518+0000 7ff830d1c700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff81006aed0 0x7ff81006d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0ca9a0 tx=0x7ff828008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.522+0000 7ff821ffb700 1 -- 172.21.15.89:0/2861672787 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff81c054530 con 0x7ff82c0b3370 2023-11-24T06:18:50.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.714+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff82c0b7c60 con 0x7ff82c0b3370 2023-11-24T06:18:50.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.715+0000 7ff821ffb700 1 -- 172.21.15.89:0/2861672787 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16482 (secure 0 0 0) 0x7ff81c057b50 con 0x7ff82c0b3370 2023-11-24T06:18:50.718 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:50.718 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:49.390863+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:50.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.717+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff81006aed0 msgr2=0x7ff81006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.717+0000 7ff833781700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff81006aed0 0x7ff81006d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0ca9a0 tx=0x7ff828008030 comp rx=0 tx=0).stop 2023-11-24T06:18:50.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.717+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 msgr2=0x7ff82c157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.717+0000 7ff833781700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c157160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00c820 tx=0x7ff81c00cbe0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.717+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 shutdown_connections 2023-11-24T06:18:50.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.718+0000 7ff833781700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff82c15bdd0 0x7ff82c0cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.718+0000 7ff833781700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff82c0b3370 0x7ff82c157160 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.718+0000 7ff833781700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff81006aed0 0x7ff81006d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.718+0000 7ff833781700 1 --2- 172.21.15.89:0/2861672787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff82c0d1c70 0x7ff82c0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.718+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 >> 172.21.15.89:0/2861672787 conn(0x7ff82c0731a0 msgr2=0x7ff82c0b0110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:50.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.718+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 shutdown_connections 2023-11-24T06:18:50.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.718+0000 7ff833781700 1 -- 172.21.15.89:0/2861672787 wait complete. 2023-11-24T06:18:50.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T06:18:50.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.852+0000 7f855e57e700 1 Processor -- start 2023-11-24T06:18:50.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.852+0000 7f855e57e700 1 -- start start 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.853+0000 7f855e57e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85580cf610 0x7f85580add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.853+0000 7f855e57e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8558073f90 0x7f8558074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.853+0000 7f855e57e700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f855806b080 con 0x7f855815f9e0 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.853+0000 7f855e57e700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8558074960 con 0x7f85580cf610 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.853+0000 7f855e57e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855815c290 con 0x7f8558073f90 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.853+0000 7f8557fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85580cf610 0x7f85580add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.853+0000 7f8557fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85580cf610 0x7f85580add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34982/0 (socket says 172.21.15.89:34982) 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.853+0000 7f8557fff700 1 -- 172.21.15.89:0/4095762976 learned_addr learned my addr 172.21.15.89:0/4095762976 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f855cb1b700 1 --2- 172.21.15.89:0/4095762976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8558073f90 0x7f8558074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f8557fff700 1 -- 172.21.15.89:0/4095762976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8558073f90 msgr2=0x7f8558074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:50.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f8557fff700 1 --2- 172.21.15.89:0/4095762976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8558073f90 0x7f8558074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f8557fff700 1 -- 172.21.15.89:0/4095762976 >> v1:172.21.15.123:6789/0 conn(0x7f855815f9e0 legacy=0x7f85580b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f8557fff700 1 -- 172.21.15.89:0/4095762976 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580cf9f0 con 0x7f85580cf610 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f8557fff700 1 --2- 172.21.15.89:0/4095762976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f85580add40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f854c0098f0 tx=0x7f854c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e888b96fae6d6963 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f8556ffd700 1 -- 172.21.15.89:0/4095762976 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c004230 con 0x7f85580cf610 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f8556ffd700 1 -- 172.21.15.89:0/4095762976 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c004390 con 0x7f85580cf610 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f8556ffd700 1 -- 172.21.15.89:0/4095762976 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c0036a0 con 0x7f85580cf610 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f855e57e700 1 -- 172.21.15.89:0/4095762976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 msgr2=0x7f85580add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f855e57e700 1 --2- 172.21.15.89:0/4095762976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f85580add40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f854c0098f0 tx=0x7f854c009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f855e57e700 1 -- 172.21.15.89:0/4095762976 shutdown_connections 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f855e57e700 1 --2- 172.21.15.89:0/4095762976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8558073f90 0x7f8558074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f855e57e700 1 --2- 172.21.15.89:0/4095762976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f85580add40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.854+0000 7f855e57e700 1 -- 172.21.15.89:0/4095762976 >> 172.21.15.89:0/4095762976 conn(0x7f85580d0bf0 msgr2=0x7f85580d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:50.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.855+0000 7f855e57e700 1 -- 172.21.15.89:0/4095762976 shutdown_connections 2023-11-24T06:18:50.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.855+0000 7f855e57e700 1 -- 172.21.15.89:0/4095762976 wait complete. 2023-11-24T06:18:50.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.856+0000 7f855e57e700 1 Processor -- start 2023-11-24T06:18:50.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.856+0000 7f855e57e700 1 -- start start 2023-11-24T06:18:50.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855e57e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8558073f90 0x7f855815e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855e57e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f855814f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855e57e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855815f9e0 0x7f8558155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855e57e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855815e4e0 con 0x7f85580cf610 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855e57e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8558150060 con 0x7f8558073f90 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855e57e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855815d8d0 con 0x7f855815f9e0 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f855814f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f855814f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34990/0 (socket says 172.21.15.89:34990) 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 -- 172.21.15.89:0/473011271 learned_addr learned my addr 172.21.15.89:0/473011271 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f8557fff700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8558073f90 0x7f855815e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855cb1b700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855815f9e0 0x7f8558155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 -- 172.21.15.89:0/473011271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855815f9e0 msgr2=0x7f8558155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855815f9e0 0x7f8558155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 -- 172.21.15.89:0/473011271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8558073f90 msgr2=0x7f855815e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8558073f90 0x7f855815e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 -- 172.21.15.89:0/473011271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c009550 con 0x7f85580cf610 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f85577fe700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f855814f530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f854800b4a0 tx=0x7f854800b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f8554ff9700 1 -- 172.21.15.89:0/473011271 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548011830 con 0x7f85580cf610 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f8554ff9700 1 -- 172.21.15.89:0/473011271 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8548011990 con 0x7f85580cf610 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855815dbb0 con 0x7f85580cf610 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f8554ff9700 1 -- 172.21.15.89:0/473011271 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548011d40 con 0x7f85580cf610 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.857+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855815e170 con 0x7f85580cf610 2023-11-24T06:18:50.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.858+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855815e690 con 0x7f85580cf610 2023-11-24T06:18:50.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.859+0000 7f8554ff9700 1 -- 172.21.15.89:0/473011271 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f854801a400 con 0x7f85580cf610 2023-11-24T06:18:50.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.859+0000 7f8554ff9700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854406aed0 0x7f854406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:50.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.859+0000 7f8554ff9700 1 -- 172.21.15.89:0/473011271 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85480892a0 con 0x7f85580cf610 2023-11-24T06:18:50.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.859+0000 7f8557fff700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854406aed0 0x7f854406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:50.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.859+0000 7f8557fff700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854406aed0 0x7f854406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f854c009130 tx=0x7f854c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:50.863+0000 7f8554ff9700 1 -- 172.21.15.89:0/473011271 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f855815e690 con 0x7f85580cf610 2023-11-24T06:18:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:51.099+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8558068fb0 con 0x7f85580cf610 2023-11-24T06:18:52.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.028+0000 7f8554ff9700 1 -- 172.21.15.89:0/473011271 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f85480542c0 con 0x7f85580cf610 2023-11-24T06:18:52.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854406aed0 msgr2=0x7f854406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:52.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854406aed0 0x7f854406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f854c009130 tx=0x7f854c008030 comp rx=0 tx=0).stop 2023-11-24T06:18:52.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 msgr2=0x7f855814f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f855814f530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f854800b4a0 tx=0x7f854800b7b0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 shutdown_connections 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855815f9e0 0x7f8558155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854406aed0 0x7f854406d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8558073f90 0x7f855815e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 --2- 172.21.15.89:0/473011271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85580cf610 0x7f855814f530 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 >> 172.21.15.89:0/473011271 conn(0x7f85580d0bf0 msgr2=0x7f85580b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 shutdown_connections 2023-11-24T06:18:52.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.031+0000 7f855e57e700 1 -- 172.21.15.89:0/473011271 wait complete. 2023-11-24T06:18:52.048 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T06:18:52.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T06:18:52.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.175+0000 7fb90b871700 1 Processor -- start 2023-11-24T06:18:52.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.175+0000 7fb90b871700 1 -- start start 2023-11-24T06:18:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb90b871700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb90415a620 0x7fb9040ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb90b871700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb90406ecf0 0x7fb9040b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb90b871700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9040b2f20 con 0x7fb9040cc540 2023-11-24T06:18:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb90b871700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9040ce690 con 0x7fb90406ecf0 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb90b871700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90415aa00 con 0x7fb90415a620 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb909e0e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb90406ecf0 0x7fb9040b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb908e0c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb9040cc540 0x7fb90406e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38500/0 (socket says 172.21.15.89:38500) 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb908e0c700 1 -- 172.21.15.89:0/1969449690 learned_addr learned my addr 172.21.15.89:0/1969449690 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb909e0e700 1 -- 172.21.15.89:0/1969449690 >> v2:172.21.15.123:3300/0 conn(0x7fb90415a620 msgr2=0x7fb9040ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb90960d700 1 --2- 172.21.15.89:0/1969449690 >> v2:172.21.15.123:3300/0 conn(0x7fb90415a620 0x7fb9040ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb909e0e700 1 --2- 172.21.15.89:0/1969449690 >> v2:172.21.15.123:3300/0 conn(0x7fb90415a620 0x7fb9040ce150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb8fbfff700 1 -- 172.21.15.89:0/1969449690 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430007728 0 0) 0x7fb9040b2f20 con 0x7fb9040cc540 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb909e0e700 1 -- 172.21.15.89:0/1969449690 >> v1:172.21.15.89:6789/0 conn(0x7fb9040cc540 legacy=0x7fb90406e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb909e0e700 1 -- 172.21.15.89:0/1969449690 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9040ce960 con 0x7fb90406ecf0 2023-11-24T06:18:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb909e0e700 1 --2- 172.21.15.89:0/1969449690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90406ecf0 0x7fb9040b3350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb904072800 tx=0x7fb8fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fef364683ee2310 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.176+0000 7fb8fbfff700 1 -- 172.21.15.89:0/1969449690 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc00e070 con 0x7fb90406ecf0 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb8fbfff700 1 -- 172.21.15.89:0/1969449690 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc00bbd0 con 0x7fb90406ecf0 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb8fbfff700 1 -- 172.21.15.89:0/1969449690 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc012430 con 0x7fb90406ecf0 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb90b871700 1 -- 172.21.15.89:0/1969449690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90406ecf0 msgr2=0x7fb9040b3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb90b871700 1 --2- 172.21.15.89:0/1969449690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90406ecf0 0x7fb9040b3350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb904072800 tx=0x7fb8fc00b330 comp rx=0 tx=0).stop 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb90b871700 1 -- 172.21.15.89:0/1969449690 shutdown_connections 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb90b871700 1 --2- 172.21.15.89:0/1969449690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90406ecf0 0x7fb9040b3350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb90b871700 1 --2- 172.21.15.89:0/1969449690 >> v2:172.21.15.123:3300/0 conn(0x7fb90415a620 0x7fb9040ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb90b871700 1 -- 172.21.15.89:0/1969449690 >> 172.21.15.89:0/1969449690 conn(0x7fb9040b2050 msgr2=0x7fb9040b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb90b871700 1 -- 172.21.15.89:0/1969449690 shutdown_connections 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.177+0000 7fb90b871700 1 -- 172.21.15.89:0/1969449690 wait complete. 2023-11-24T06:18:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.178+0000 7fb90b871700 1 Processor -- start 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.178+0000 7fb90b871700 1 -- start start 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb90b871700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb90406ecf0 0x7fb90415c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb90b871700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9040cc540 0x7fb904153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb90b871700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90415a620 0x7fb904154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb90b871700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90414b710 con 0x7fb90415a620 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb90b871700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9041f3040 con 0x7fb9040cc540 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb90b871700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9041f3320 con 0x7fb90406ecf0 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb909e0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90415a620 0x7fb904154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb909e0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90415a620 0x7fb904154490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35004/0 (socket says 172.21.15.89:35004) 2023-11-24T06:18:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb909e0e700 1 -- 172.21.15.89:0/2202730625 learned_addr learned my addr 172.21.15.89:0/2202730625 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb908e0c700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9040cc540 0x7fb904153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb90960d700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb90406ecf0 0x7fb90415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.179+0000 7fb909e0e700 1 -- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb90406ecf0 msgr2=0x7fb90415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb909e0e700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb90406ecf0 0x7fb90415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb909e0e700 1 -- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9040cc540 msgr2=0x7fb904153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb909e0e700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9040cc540 0x7fb904153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb909e0e700 1 -- 172.21.15.89:0/2202730625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc009a70 con 0x7fb90415a620 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb908e0c700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9040cc540 0x7fb904153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb909e0e700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90415a620 0x7fb904154490 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb904070e40 tx=0x7fb8fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb8f9ffb700 1 -- 172.21.15.89:0/2202730625 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc012430 con 0x7fb90415a620 2023-11-24T06:18:52.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb8f9ffb700 1 -- 172.21.15.89:0/2202730625 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc020070 con 0x7fb90415a620 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9041f3600 con 0x7fb90415a620 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb8f9ffb700 1 -- 172.21.15.89:0/2202730625 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc004280 con 0x7fb90415a620 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.180+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9041f3bc0 con 0x7fb90415a620 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.181+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb904068d90 con 0x7fb90415a620 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.181+0000 7fb8f9ffb700 1 -- 172.21.15.89:0/2202730625 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb8fc0043e0 con 0x7fb90415a620 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.182+0000 7fb8f9ffb700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8e806aed0 0x7fb8e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.182+0000 7fb8f9ffb700 1 -- 172.21.15.89:0/2202730625 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8fc022070 con 0x7fb90415a620 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.182+0000 7fb90960d700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8e806aed0 0x7fb8e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:52.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.182+0000 7fb90960d700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8e806aed0 0x7fb8e806d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb90000a950 tx=0x7fb900008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:52.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.186+0000 7fb8f9ffb700 1 -- 172.21.15.89:0/2202730625 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8fc056980 con 0x7fb90415a620 2023-11-24T06:18:52.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:52.421+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb9041f3d50 con 0x7fb90415a620 2023-11-24T06:18:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.029+0000 7fb8f9ffb700 1 -- 172.21.15.89:0/2202730625 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb8fc01ae60 con 0x7fb90415a620 2023-11-24T06:18:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8e806aed0 msgr2=0x7fb8e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8e806aed0 0x7fb8e806d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb90000a950 tx=0x7fb900008040 comp rx=0 tx=0).stop 2023-11-24T06:18:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90415a620 msgr2=0x7fb904154490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90415a620 0x7fb904154490 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb904070e40 tx=0x7fb8fc004ad0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 shutdown_connections 2023-11-24T06:18:53.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb90415a620 0x7fb904154490 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb90406ecf0 0x7fb90415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8e806aed0 0x7fb8e806d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 --2- 172.21.15.89:0/2202730625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9040cc540 0x7fb904153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 >> 172.21.15.89:0/2202730625 conn(0x7fb9040b2050 msgr2=0x7fb9040b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:53.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 shutdown_connections 2023-11-24T06:18:53.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.032+0000 7fb90b871700 1 -- 172.21.15.89:0/2202730625 wait complete. 2023-11-24T06:18:53.048 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T06:18:53.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:18:53.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.169+0000 7f7ebaf21700 1 Processor -- start 2023-11-24T06:18:53.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.169+0000 7f7ebaf21700 1 -- start start 2023-11-24T06:18:53.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.170+0000 7f7ebaf21700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7eb40b59f0 0x7f7eb406db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.170+0000 7f7ebaf21700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb40b4820 con 0x7f7eb406e0b0 2023-11-24T06:18:53.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.170+0000 7f7ebaf21700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb40b4b00 con 0x7f7eb40b4350 2023-11-24T06:18:53.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.170+0000 7f7ebaf21700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb40ae9b0 con 0x7f7eb40b59f0 2023-11-24T06:18:53.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.170+0000 7f7eb3fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7eb406e0b0 0x7f7eb40b3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38514/0 (socket says 172.21.15.89:38514) 2023-11-24T06:18:53.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.170+0000 7f7eb3fff700 1 -- 172.21.15.89:0/371430135 learned_addr learned my addr 172.21.15.89:0/371430135 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:53.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.170+0000 7f7eb8cbd700 1 --2- 172.21.15.89:0/371430135 >> v2:172.21.15.123:3300/0 conn(0x7f7eb40b59f0 0x7f7eb406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.170+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332506266 0 0) 0x7f7eb40b4820 con 0x7f7eb406e0b0 2023-11-24T06:18:53.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea40035b0 con 0x7f7eb406e0b0 2023-11-24T06:18:53.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643490521 0 0) 0x7f7ea40035b0 con 0x7f7eb406e0b0 2023-11-24T06:18:53.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 >> v2:172.21.15.123:3300/0 conn(0x7f7eb40b59f0 msgr2=0x7f7eb406db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:53.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 --2- 172.21.15.89:0/371430135 >> v2:172.21.15.123:3300/0 conn(0x7f7eb40b59f0 0x7f7eb406db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 >> v1:172.21.15.123:6789/0 conn(0x7f7eb40b4350 legacy=0x7f7eb40af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb40aec90 con 0x7f7eb406e0b0 2023-11-24T06:18:53.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f7ea0001670 con 0x7f7eb406e0b0 2023-11-24T06:18:53.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 >> v1:172.21.15.89:6789/0 conn(0x7f7eb406e0b0 legacy=0x7f7eb40b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4003a00 0x7f7ea4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7ea4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ea4005210 0x7f7ea400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40ae9b0 con 0x7f7ea4004850 2023-11-24T06:18:53.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea400c1b0 con 0x7f7ea4003a00 2023-11-24T06:18:53.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea400c370 con 0x7f7ea4005210 2023-11-24T06:18:53.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb3fff700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7ea4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb8cbd700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4003a00 0x7f7ea4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb94be700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ea4005210 0x7f7ea400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb3fff700 1 -- 172.21.15.89:0/371430135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ea4005210 msgr2=0x7f7ea400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:53.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb3fff700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ea4005210 0x7f7ea400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb3fff700 1 -- 172.21.15.89:0/371430135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4003a00 msgr2=0x7f7ea4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:53.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb3fff700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4003a00 0x7f7ea4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb94be700 1 -- 172.21.15.89:0/371430135 reap_dead start 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb3fff700 1 -- 172.21.15.89:0/371430135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea40035b0 con 0x7f7ea4004850 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb3fff700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7ea4004cd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000c270 tx=0x7f7ea4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9fb194f20bf881 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.171+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0007c10 con 0x7f7ea4004850 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea000cd50 con 0x7f7ea4004850 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7eb37fe700 1 -- 172.21.15.89:0/371430135 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0005ba0 con 0x7f7ea4004850 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7ebaf21700 1 -- 172.21.15.89:0/371430135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 msgr2=0x7f7ea4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7ebaf21700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7ea4004cd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000c270 tx=0x7f7ea4005880 comp rx=0 tx=0).stop 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7ebaf21700 1 -- 172.21.15.89:0/371430135 shutdown_connections 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7ebaf21700 1 --2- 172.21.15.89:0/371430135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7ea4004cd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7ebaf21700 1 -- 172.21.15.89:0/371430135 >> 172.21.15.89:0/371430135 conn(0x7f7eb4072990 msgr2=0x7f7eb406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:53.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7ebaf21700 1 -- 172.21.15.89:0/371430135 shutdown_connections 2023-11-24T06:18:53.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.172+0000 7f7ebaf21700 1 -- 172.21.15.89:0/371430135 wait complete. 2023-11-24T06:18:53.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.173+0000 7f7ebaf21700 1 Processor -- start 2023-11-24T06:18:53.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.173+0000 7f7ebaf21700 1 -- start start 2023-11-24T06:18:53.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7ebaf21700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7eb41466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7ebaf21700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb4154d80 0x7f7eb41f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7ebaf21700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eb4152950 0x7f7eb4152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7ebaf21700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4153310 con 0x7f7ea4004850 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7ebaf21700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb414fae0 con 0x7f7eb4154d80 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7ebaf21700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb41548f0 con 0x7f7eb4152950 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7eb8cbd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7eb41466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7eb8cbd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7eb41466d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35020/0 (socket says 172.21.15.89:35020) 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7eb8cbd700 1 -- 172.21.15.89:0/158944646 learned_addr learned my addr 172.21.15.89:0/158944646 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.174+0000 7f7eb3fff700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb4154d80 0x7f7eb41f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb94be700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eb4152950 0x7f7eb4152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb8cbd700 1 -- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eb4152950 msgr2=0x7f7eb4152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb8cbd700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eb4152950 0x7f7eb4152dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb8cbd700 1 -- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb4154d80 msgr2=0x7f7eb41f4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb8cbd700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb4154d80 0x7f7eb41f4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb8cbd700 1 -- 172.21.15.89:0/158944646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4146c10 con 0x7f7ea4004850 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb3fff700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb4154d80 0x7f7eb41f4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb94be700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eb4152950 0x7f7eb4152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb8cbd700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7eb41466d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8011b00 tx=0x7f7ea8011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb17fa700 1 -- 172.21.15.89:0/158944646 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea800f9a0 con 0x7f7ea4004850 2023-11-24T06:18:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb17fa700 1 -- 172.21.15.89:0/158944646 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea800fb20 con 0x7f7ea4004850 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb4153a00 con 0x7f7ea4004850 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7eb17fa700 1 -- 172.21.15.89:0/158944646 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea800fed0 con 0x7f7ea4004850 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.175+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb4153fc0 con 0x7f7ea4004850 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.176+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb4067f60 con 0x7f7ea4004850 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.176+0000 7f7eb17fa700 1 -- 172.21.15.89:0/158944646 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7ea8014440 con 0x7f7ea4004850 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.177+0000 7f7eb17fa700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9806aed0 0x7f7e9806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.177+0000 7f7eb17fa700 1 -- 172.21.15.89:0/158944646 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ea8089e00 con 0x7f7ea4004850 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.177+0000 7f7eb3fff700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9806aed0 0x7f7e9806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.177+0000 7f7eb3fff700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9806aed0 0x7f7e9806d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7eb41618e0 tx=0x7f7ea0007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.181+0000 7f7eb17fa700 1 -- 172.21.15.89:0/158944646 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ea8054f70 con 0x7f7ea4004850 2023-11-24T06:18:53.320 INFO:tasks.ceph.mds.d.smithi123.stderr:2023-11-24T06:18:53.318+0000 7ff5b25c2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:18:53.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.416+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7eb4154bd0 con 0x7f7ea4004850 2023-11-24T06:18:53.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.416+0000 7f7eb17fa700 1 -- 172.21.15.89:0/158944646 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f7ea8058590 con 0x7f7ea4004850 2023-11-24T06:18:53.421 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:53.421 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:18:49.391199+0000","modified":"2023-11-24T06:18:52.422301+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:18:53.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9806aed0 msgr2=0x7f7e9806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9806aed0 0x7f7e9806d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7eb41618e0 tx=0x7f7ea0007000 comp rx=0 tx=0).stop 2023-11-24T06:18:53.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 msgr2=0x7f7eb41466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7eb41466d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8011b00 tx=0x7f7ea8011ec0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 shutdown_connections 2023-11-24T06:18:53.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eb4152950 0x7f7eb4152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ea4004850 0x7f7eb41466d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9806aed0 0x7f7e9806d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 --2- 172.21.15.89:0/158944646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb4154d80 0x7f7eb41f4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 >> 172.21.15.89:0/158944646 conn(0x7f7eb4072990 msgr2=0x7f7eb406e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:53.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 shutdown_connections 2023-11-24T06:18:53.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.419+0000 7f7ebaf21700 1 -- 172.21.15.89:0/158944646 wait complete. 2023-11-24T06:18:53.430 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-11-24T06:18:53.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:53.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.554+0000 7fd12a66e700 1 Processor -- start 2023-11-24T06:18:53.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.554+0000 7fd12a66e700 1 -- start start 2023-11-24T06:18:53.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd12a66e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd12415e080 0x7fd1240b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd12a66e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1240af4d0 0x7fd1240afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd12a66e700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1240ae430 con 0x7fd12406c000 2023-11-24T06:18:53.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd12a66e700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1240af0b0 con 0x7fd1240af4d0 2023-11-24T06:18:53.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd12a66e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1240b05a0 con 0x7fd12415e080 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd1237fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1240af4d0 0x7fd1240afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd128c0b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd12406c000 0x7fd124073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38536/0 (socket says 172.21.15.89:38536) 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd1237fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1240af4d0 0x7fd1240afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35028/0 (socket says 172.21.15.89:35028) 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd128c0b700 1 -- 172.21.15.89:0/867610194 learned_addr learned my addr 172.21.15.89:0/867610194 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd123fff700 1 --2- 172.21.15.89:0/867610194 >> v2:172.21.15.123:3300/0 conn(0x7fd12415e080 0x7fd1240b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd1237fe700 1 -- 172.21.15.89:0/867610194 >> v2:172.21.15.123:3300/0 conn(0x7fd12415e080 msgr2=0x7fd1240b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd1237fe700 1 --2- 172.21.15.89:0/867610194 >> v2:172.21.15.123:3300/0 conn(0x7fd12415e080 0x7fd1240b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd1237fe700 1 -- 172.21.15.89:0/867610194 >> v1:172.21.15.89:6789/0 conn(0x7fd12406c000 legacy=0x7fd124073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd1237fe700 1 -- 172.21.15.89:0/867610194 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1240ae710 con 0x7fd1240af4d0 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd1237fe700 1 --2- 172.21.15.89:0/867610194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1240af4d0 0x7fd1240afb90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd124072a90 tx=0x7fd114009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e822fa3c802d0b3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd122ffd700 1 -- 172.21.15.89:0/867610194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114011830 con 0x7fd1240af4d0 2023-11-24T06:18:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd122ffd700 1 -- 172.21.15.89:0/867610194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd114011990 con 0x7fd1240af4d0 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.555+0000 7fd122ffd700 1 -- 172.21.15.89:0/867610194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114011d20 con 0x7fd1240af4d0 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.556+0000 7fd12a66e700 1 -- 172.21.15.89:0/867610194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1240af4d0 msgr2=0x7fd1240afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.556+0000 7fd12a66e700 1 --2- 172.21.15.89:0/867610194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1240af4d0 0x7fd1240afb90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd124072a90 tx=0x7fd114009b50 comp rx=0 tx=0).stop 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.556+0000 7fd12a66e700 1 -- 172.21.15.89:0/867610194 shutdown_connections 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.556+0000 7fd12a66e700 1 --2- 172.21.15.89:0/867610194 >> v2:172.21.15.123:3300/0 conn(0x7fd12415e080 0x7fd1240b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.556+0000 7fd12a66e700 1 --2- 172.21.15.89:0/867610194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1240af4d0 0x7fd1240afb90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.556+0000 7fd12a66e700 1 -- 172.21.15.89:0/867610194 >> 172.21.15.89:0/867610194 conn(0x7fd124071b90 msgr2=0x7fd124071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.556+0000 7fd12a66e700 1 -- 172.21.15.89:0/867610194 shutdown_connections 2023-11-24T06:18:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.556+0000 7fd12a66e700 1 -- 172.21.15.89:0/867610194 wait complete. 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.557+0000 7fd12a66e700 1 Processor -- start 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.557+0000 7fd12a66e700 1 -- start start 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd12a66e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12406c000 0x7fd124152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd12a66e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1240af4d0 0x7fd124157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd12a66e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd12415e080 0x7fd1241453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd12a66e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12415a2f0 con 0x7fd12406c000 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd12a66e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd124151c10 con 0x7fd1240af4d0 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd12a66e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12415eb60 con 0x7fd12415e080 2023-11-24T06:18:53.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd123fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12406c000 0x7fd124152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd123fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12406c000 0x7fd124152190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35034/0 (socket says 172.21.15.89:35034) 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd123fff700 1 -- 172.21.15.89:0/856929027 learned_addr learned my addr 172.21.15.89:0/856929027 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd1237fe700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1240af4d0 0x7fd124157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd128c0b700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd12415e080 0x7fd1241453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd123fff700 1 -- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd12415e080 msgr2=0x7fd1241453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd123fff700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd12415e080 0x7fd1241453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd123fff700 1 -- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1240af4d0 msgr2=0x7fd124157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd123fff700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1240af4d0 0x7fd124157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd123fff700 1 -- 172.21.15.89:0/856929027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1140094a0 con 0x7fd12406c000 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.558+0000 7fd128c0b700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd12415e080 0x7fd1241453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.559+0000 7fd123fff700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12406c000 0x7fd124152190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd11800c7f0 tx=0x7fd11800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.559+0000 7fd120ff9700 1 -- 172.21.15.89:0/856929027 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118011800 con 0x7fd12406c000 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.559+0000 7fd120ff9700 1 -- 172.21.15.89:0/856929027 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd118011960 con 0x7fd12406c000 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.559+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1241526d0 con 0x7fd12406c000 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.559+0000 7fd120ff9700 1 -- 172.21.15.89:0/856929027 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118011cd0 con 0x7fd12406c000 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.559+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12415f0e0 con 0x7fd12406c000 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.560+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1240753f0 con 0x7fd12406c000 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.560+0000 7fd120ff9700 1 -- 172.21.15.89:0/856929027 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd118010350 con 0x7fd12406c000 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.561+0000 7fd120ff9700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11006aed0 0x7fd11006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.561+0000 7fd1237fe700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11006aed0 0x7fd11006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.561+0000 7fd120ff9700 1 -- 172.21.15.89:0/856929027 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd118089170 con 0x7fd12406c000 2023-11-24T06:18:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.561+0000 7fd1237fe700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11006aed0 0x7fd11006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd124072a90 tx=0x7fd114008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.565+0000 7fd120ff9700 1 -- 172.21.15.89:0/856929027 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd118054230 con 0x7fd12406c000 2023-11-24T06:18:53.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.757+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1240d09b0 con 0x7fd12406c000 2023-11-24T06:18:53.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.758+0000 7fd120ff9700 1 -- 172.21.15.89:0/856929027 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7fd118057850 con 0x7fd12406c000 2023-11-24T06:18:53.761 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:53.761 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:53.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.760+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11006aed0 msgr2=0x7fd11006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.760+0000 7fd12a66e700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11006aed0 0x7fd11006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd124072a90 tx=0x7fd114008030 comp rx=0 tx=0).stop 2023-11-24T06:18:53.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.760+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12406c000 msgr2=0x7fd124152190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.760+0000 7fd12a66e700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12406c000 0x7fd124152190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd11800c7f0 tx=0x7fd11800cb00 comp rx=0 tx=0).stop 2023-11-24T06:18:53.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.761+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 shutdown_connections 2023-11-24T06:18:53.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.761+0000 7fd12a66e700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd12415e080 0x7fd1241453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.761+0000 7fd12a66e700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12406c000 0x7fd124152190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.761+0000 7fd12a66e700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11006aed0 0x7fd11006d390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.761+0000 7fd12a66e700 1 --2- 172.21.15.89:0/856929027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1240af4d0 0x7fd124157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.761+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 >> 172.21.15.89:0/856929027 conn(0x7fd124071b90 msgr2=0x7fd12415d290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:53.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.761+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 shutdown_connections 2023-11-24T06:18:53.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.761+0000 7fd12a66e700 1 -- 172.21.15.89:0/856929027 wait complete. 2023-11-24T06:18:53.775 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T06:18:53.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:18:53.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.893+0000 7fce9a36d700 1 Processor -- start 2023-11-24T06:18:53.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.894+0000 7fce9a36d700 1 -- start start 2023-11-24T06:18:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.894+0000 7fce9a36d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fce9415e080 0x7fce940b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.894+0000 7fce9a36d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fce9406c710 0x7fce940afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.894+0000 7fce9a36d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce940ae430 con 0x7fce940af4d0 2023-11-24T06:18:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.894+0000 7fce9a36d700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce940af0b0 con 0x7fce9415e080 2023-11-24T06:18:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.894+0000 7fce9a36d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce940b05a0 con 0x7fce9406c710 2023-11-24T06:18:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce93fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fce9415e080 0x7fce940b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce93fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fce9415e080 0x7fce940b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35042/0 (socket says 172.21.15.89:35042) 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce93fff700 1 -- 172.21.15.89:0/2400649112 learned_addr learned my addr 172.21.15.89:0/2400649112 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce937fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fce940af4d0 0x7fce9406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38546/0 (socket says 172.21.15.89:38546) 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce9890a700 1 --2- 172.21.15.89:0/2400649112 >> v2:172.21.15.123:3300/0 conn(0x7fce9406c710 0x7fce940afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce93fff700 1 -- 172.21.15.89:0/2400649112 >> v2:172.21.15.123:3300/0 conn(0x7fce9406c710 msgr2=0x7fce940afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce93fff700 1 --2- 172.21.15.89:0/2400649112 >> v2:172.21.15.123:3300/0 conn(0x7fce9406c710 0x7fce940afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce93fff700 1 -- 172.21.15.89:0/2400649112 >> v1:172.21.15.89:6789/0 conn(0x7fce940af4d0 legacy=0x7fce9406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce93fff700 1 -- 172.21.15.89:0/2400649112 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce940ae710 con 0x7fce9415e080 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce92ffd700 1 -- 172.21.15.89:0/2400649112 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249841243 0 0) 0x7fce940ae430 con 0x7fce940af4d0 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce93fff700 1 --2- 172.21.15.89:0/2400649112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce9415e080 0x7fce940b5050 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fce9406e060 tx=0x7fce84009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8e2200527ed2cf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce92ffd700 1 -- 172.21.15.89:0/2400649112 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84011830 con 0x7fce9415e080 2023-11-24T06:18:53.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce92ffd700 1 -- 172.21.15.89:0/2400649112 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce84011990 con 0x7fce9415e080 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.895+0000 7fce92ffd700 1 -- 172.21.15.89:0/2400649112 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84011d20 con 0x7fce9415e080 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.896+0000 7fce9a36d700 1 -- 172.21.15.89:0/2400649112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce9415e080 msgr2=0x7fce940b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.896+0000 7fce9a36d700 1 --2- 172.21.15.89:0/2400649112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce9415e080 0x7fce940b5050 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fce9406e060 tx=0x7fce84009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.896+0000 7fce9a36d700 1 -- 172.21.15.89:0/2400649112 shutdown_connections 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.896+0000 7fce9a36d700 1 --2- 172.21.15.89:0/2400649112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce9415e080 0x7fce940b5050 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.896+0000 7fce9a36d700 1 --2- 172.21.15.89:0/2400649112 >> v2:172.21.15.123:3300/0 conn(0x7fce9406c710 0x7fce940afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.896+0000 7fce9a36d700 1 -- 172.21.15.89:0/2400649112 >> 172.21.15.89:0/2400649112 conn(0x7fce94071b90 msgr2=0x7fce94071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.896+0000 7fce9a36d700 1 -- 172.21.15.89:0/2400649112 shutdown_connections 2023-11-24T06:18:53.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.896+0000 7fce9a36d700 1 -- 172.21.15.89:0/2400649112 wait complete. 2023-11-24T06:18:53.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.897+0000 7fce9a36d700 1 Processor -- start 2023-11-24T06:18:53.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.897+0000 7fce9a36d700 1 -- start start 2023-11-24T06:18:53.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce9a36d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce9406c710 0x7fce94154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce9a36d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce940af4d0 0x7fce94159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce9a36d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce9415e080 0x7fce94153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce9a36d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9415a220 con 0x7fce940af4d0 2023-11-24T06:18:53.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce9a36d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce94155eb0 con 0x7fce9415e080 2023-11-24T06:18:53.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce9a36d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce94156340 con 0x7fce9406c710 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce940af4d0 0x7fce94159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce940af4d0 0x7fce94159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35050/0 (socket says 172.21.15.89:35050) 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 -- 172.21.15.89:0/3185071390 learned_addr learned my addr 172.21.15.89:0/3185071390 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 -- 172.21.15.89:0/3185071390 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce9406c710 msgr2=0x7fce94154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce93fff700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce9406c710 0x7fce94154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce9406c710 0x7fce94154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 -- 172.21.15.89:0/3185071390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce9415e080 msgr2=0x7fce94153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce9890a700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce9415e080 0x7fce94153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce9415e080 0x7fce94153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 -- 172.21.15.89:0/3185071390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84009550 con 0x7fce940af4d0 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.898+0000 7fce937fe700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce940af4d0 0x7fce94159ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fce8800c540 tx=0x7fce8800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.899+0000 7fce90ff9700 1 -- 172.21.15.89:0/3185071390 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88004c80 con 0x7fce940af4d0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.899+0000 7fce90ff9700 1 -- 172.21.15.89:0/3185071390 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce88004de0 con 0x7fce940af4d0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.899+0000 7fce90ff9700 1 -- 172.21.15.89:0/3185071390 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88007310 con 0x7fce940af4d0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.899+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce941552a0 con 0x7fce940af4d0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.899+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce94155860 con 0x7fce940af4d0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.899+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce940753f0 con 0x7fce940af4d0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.900+0000 7fce90ff9700 1 -- 172.21.15.89:0/3185071390 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fce88007470 con 0x7fce940af4d0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.900+0000 7fce90ff9700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce8006aed0 0x7fce8006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.900+0000 7fce90ff9700 1 -- 172.21.15.89:0/3185071390 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce88089330 con 0x7fce940af4d0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.900+0000 7fce93fff700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce8006aed0 0x7fce8006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:53.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.901+0000 7fce93fff700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce8006aed0 0x7fce8006d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fce84009870 tx=0x7fce84008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:53.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:53.905+0000 7fce90ff9700 1 -- 172.21.15.89:0/3185071390 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce880543e0 con 0x7fce940af4d0 2023-11-24T06:18:54.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.092+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce940aecd0 con 0x7fce8006aed0 2023-11-24T06:18:54.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.100+0000 7fce90ff9700 1 -- 172.21.15.89:0/3185071390 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264659 (secure 0 0 0) 0x7fce8401f070 con 0x7fce8006aed0 2023-11-24T06:18:54.105 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:54.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.103+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce8006aed0 msgr2=0x7fce8006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.103+0000 7fce9a36d700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce8006aed0 0x7fce8006d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fce84009870 tx=0x7fce84008030 comp rx=0 tx=0).stop 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.103+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce940af4d0 msgr2=0x7fce94159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.103+0000 7fce9a36d700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce940af4d0 0x7fce94159ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fce8800c540 tx=0x7fce8800c900 comp rx=0 tx=0).stop 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.103+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 shutdown_connections 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.103+0000 7fce9a36d700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce8006aed0 0x7fce8006d390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.104+0000 7fce9a36d700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce9415e080 0x7fce94153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.104+0000 7fce9a36d700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce9406c710 0x7fce94154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.104+0000 7fce9a36d700 1 --2- 172.21.15.89:0/3185071390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce940af4d0 0x7fce94159ce0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.104+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 >> 172.21.15.89:0/3185071390 conn(0x7fce94071b90 msgr2=0x7fce9415d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.104+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 shutdown_connections 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.104+0000 7fce9a36d700 1 -- 172.21.15.89:0/3185071390 wait complete. 2023-11-24T06:18:54.108 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:18:54.117 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-11-24T06:18:53.718677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2848,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2916352,"data_stored":1752404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10],"upper_bound":16},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.730641"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024214+0000","last_change":"2023-11-24T06:18:50.033723+0000","last_active":"2023-11-24T06:18:51.024214+0000","last_peered":"2023-11-24T06:18:51.024214+0000","last_clean":"2023-11-24T06:18:51.024214+0000","last_became_active":"2023-11-24T06:18:50.033433+0000","last_became_peered":"2023-11-24T06:18:50.033433+0000","last_unstale":"2023-11-24T06:18:51.024214+0000","last_undegraded":"2023-11-24T06:18:51.024214+0000","last_fullsized":"2023-11-24T06:18:51.024214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:49.822079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021557+0000","last_change":"2023-11-24T06:18:50.024643+0000","last_active":"2023-11-24T06:18:51.021557+0000","last_peered":"2023-11-24T06:18:51.021557+0000","last_clean":"2023-11-24T06:18:51.021557+0000","last_became_active":"2023-11-24T06:18:50.024352+0000","last_became_peered":"2023-11-24T06:18:50.024352+0000","last_unstale":"2023-11-24T06:18:51.021557+0000","last_undegraded":"2023-11-24T06:18:51.021557+0000","last_fullsized":"2023-11-24T06:18:51.021557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:59.327185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032185+0000","last_change":"2023-11-24T06:18:50.025637+0000","last_active":"2023-11-24T06:18:50.032185+0000","last_peered":"2023-11-24T06:18:50.032185+0000","last_clean":"2023-11-24T06:18:50.032185+0000","last_became_active":"2023-11-24T06:18:50.025328+0000","last_became_peered":"2023-11-24T06:18:50.025328+0000","last_unstale":"2023-11-24T06:18:50.032185+0000","last_undegraded":"2023-11-24T06:18:50.032185+0000","last_fullsized":"2023-11-24T06:18:50.032185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.039592+0000","last_change":"2023-11-24T06:18:50.030667+0000","last_active":"2023-11-24T06:18:50.039592+0000","last_peered":"2023-11-24T06:18:50.039592+0000","last_clean":"2023-11-24T06:18:50.039592+0000","last_became_active":"2023-11-24T06:18:50.030133+0000","last_became_peered":"2023-11-24T06:18:50.030133+0000","last_unstale":"2023-11-24T06:18:50.039592+0000","last_undegraded":"2023-11-24T06:18:50.039592+0000","last_fullsized":"2023-11-24T06:18:50.039592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024116+0000","last_change":"2023-11-24T06:18:50.025413+0000","last_active":"2023-11-24T06:18:51.024116+0000","last_peered":"2023-11-24T06:18:51.024116+0000","last_clean":"2023-11-24T06:18:51.024116+0000","last_became_active":"2023-11-24T06:18:50.025118+0000","last_became_peered":"2023-11-24T06:18:50.025118+0000","last_unstale":"2023-11-24T06:18:51.024116+0000","last_undegraded":"2023-11-24T06:18:51.024116+0000","last_fullsized":"2023-11-24T06:18:51.024116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:21.299524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041485+0000","last_change":"2023-11-24T06:18:50.039892+0000","last_active":"2023-11-24T06:18:50.041485+0000","last_peered":"2023-11-24T06:18:50.041485+0000","last_clean":"2023-11-24T06:18:50.041485+0000","last_became_active":"2023-11-24T06:18:50.039525+0000","last_became_peered":"2023-11-24T06:18:50.039525+0000","last_unstale":"2023-11-24T06:18:50.041485+0000","last_undegraded":"2023-11-24T06:18:50.041485+0000","last_fullsized":"2023-11-24T06:18:50.041485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:41.341947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022843+0000","last_change":"2023-11-24T06:18:50.024213+0000","last_active":"2023-11-24T06:18:51.022843+0000","last_peered":"2023-11-24T06:18:51.022843+0000","last_clean":"2023-11-24T06:18:51.022843+0000","last_became_active":"2023-11-24T06:18:50.023914+0000","last_became_peered":"2023-11-24T06:18:50.023914+0000","last_unstale":"2023-11-24T06:18:51.022843+0000","last_undegraded":"2023-11-24T06:18:51.022843+0000","last_fullsized":"2023-11-24T06:18:51.022843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:48.527966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038582+0000","last_change":"2023-11-24T06:18:50.037907+0000","last_active":"2023-11-24T06:18:50.038582+0000","last_peered":"2023-11-24T06:18:50.038582+0000","last_clean":"2023-11-24T06:18:50.038582+0000","last_became_active":"2023-11-24T06:18:50.037606+0000","last_became_peered":"2023-11-24T06:18:50.037606+0000","last_unstale":"2023-11-24T06:18:50.038582+0000","last_undegraded":"2023-11-24T06:18:50.038582+0000","last_fullsized":"2023-11-24T06:18:50.038582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024189+0000","last_change":"2023-11-24T06:18:50.035382+0000","last_active":"2023-11-24T06:18:51.024189+0000","last_peered":"2023-11-24T06:18:51.024189+0000","last_clean":"2023-11-24T06:18:51.024189+0000","last_became_active":"2023-11-24T06:18:50.035148+0000","last_became_peered":"2023-11-24T06:18:50.035148+0000","last_unstale":"2023-11-24T06:18:51.024189+0000","last_undegraded":"2023-11-24T06:18:51.024189+0000","last_fullsized":"2023-11-24T06:18:51.024189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:39.141407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.039953+0000","last_change":"2023-11-24T06:18:50.037425+0000","last_active":"2023-11-24T06:18:50.039953+0000","last_peered":"2023-11-24T06:18:50.039953+0000","last_clean":"2023-11-24T06:18:50.039953+0000","last_became_active":"2023-11-24T06:18:50.037111+0000","last_became_peered":"2023-11-24T06:18:50.037111+0000","last_unstale":"2023-11-24T06:18:50.039953+0000","last_undegraded":"2023-11-24T06:18:50.039953+0000","last_fullsized":"2023-11-24T06:18:50.039953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:25.578418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040281+0000","last_change":"2023-11-24T06:18:50.036955+0000","last_active":"2023-11-24T06:18:50.040281+0000","last_peered":"2023-11-24T06:18:50.040281+0000","last_clean":"2023-11-24T06:18:50.040281+0000","last_became_active":"2023-11-24T06:18:50.036643+0000","last_became_peered":"2023-11-24T06:18:50.036643+0000","last_unstale":"2023-11-24T06:18:50.040281+0000","last_undegraded":"2023-11-24T06:18:50.040281+0000","last_fullsized":"2023-11-24T06:18:50.040281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:41.341947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023821+0000","last_change":"2023-11-24T06:18:50.034218+0000","last_active":"2023-11-24T06:18:51.023821+0000","last_peered":"2023-11-24T06:18:51.023821+0000","last_clean":"2023-11-24T06:18:51.023821+0000","last_became_active":"2023-11-24T06:18:50.033754+0000","last_became_peered":"2023-11-24T06:18:50.033754+0000","last_unstale":"2023-11-24T06:18:51.023821+0000","last_undegraded":"2023-11-24T06:18:51.023821+0000","last_fullsized":"2023-11-24T06:18:51.023821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036464+0000","last_change":"2023-11-24T06:18:50.033709+0000","last_active":"2023-11-24T06:18:50.036464+0000","last_peered":"2023-11-24T06:18:50.036464+0000","last_clean":"2023-11-24T06:18:50.036464+0000","last_became_active":"2023-11-24T06:18:50.033447+0000","last_became_peered":"2023-11-24T06:18:50.033447+0000","last_unstale":"2023-11-24T06:18:50.036464+0000","last_undegraded":"2023-11-24T06:18:50.036464+0000","last_fullsized":"2023-11-24T06:18:50.036464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:25.578418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.031860+0000","last_change":"2023-11-24T06:18:50.023134+0000","last_active":"2023-11-24T06:18:50.031860+0000","last_peered":"2023-11-24T06:18:50.031860+0000","last_clean":"2023-11-24T06:18:50.031860+0000","last_became_active":"2023-11-24T06:18:50.022837+0000","last_became_peered":"2023-11-24T06:18:50.022837+0000","last_unstale":"2023-11-24T06:18:50.031860+0000","last_undegraded":"2023-11-24T06:18:50.031860+0000","last_fullsized":"2023-11-24T06:18:50.031860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:27.849578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021102+0000","last_change":"2023-11-24T06:18:50.027089+0000","last_active":"2023-11-24T06:18:51.021102+0000","last_peered":"2023-11-24T06:18:51.021102+0000","last_clean":"2023-11-24T06:18:51.021102+0000","last_became_active":"2023-11-24T06:18:50.026810+0000","last_became_peered":"2023-11-24T06:18:50.026810+0000","last_unstale":"2023-11-24T06:18:51.021102+0000","last_undegraded":"2023-11-24T06:18:51.021102+0000","last_fullsized":"2023-11-24T06:18:51.021102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035433+0000","last_change":"2023-11-24T06:18:50.028542+0000","last_active":"2023-11-24T06:18:50.035433+0000","last_peered":"2023-11-24T06:18:50.035433+0000","last_clean":"2023-11-24T06:18:50.035433+0000","last_became_active":"2023-11-24T06:18:50.027971+0000","last_became_peered":"2023-11-24T06:18:50.027971+0000","last_unstale":"2023-11-24T06:18:50.035433+0000","last_undegraded":"2023-11-24T06:18:50.035433+0000","last_fullsized":"2023-11-24T06:18:50.035433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:27.849578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038210+0000","last_change":"2023-11-24T06:18:50.035313+0000","last_active":"2023-11-24T06:18:50.038210+0000","last_peered":"2023-11-24T06:18:50.038210+0000","last_clean":"2023-11-24T06:18:50.038210+0000","last_became_active":"2023-11-24T06:18:50.034744+0000","last_became_peered":"2023-11-24T06:18:50.034744+0000","last_unstale":"2023-11-24T06:18:50.038210+0000","last_undegraded":"2023-11-24T06:18:50.038210+0000","last_fullsized":"2023-11-24T06:18:50.038210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:34.485750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036447+0000","last_change":"2023-11-24T06:18:50.034072+0000","last_active":"2023-11-24T06:18:50.036447+0000","last_peered":"2023-11-24T06:18:50.036447+0000","last_clean":"2023-11-24T06:18:50.036447+0000","last_became_active":"2023-11-24T06:18:50.033798+0000","last_became_peered":"2023-11-24T06:18:50.033798+0000","last_unstale":"2023-11-24T06:18:50.036447+0000","last_undegraded":"2023-11-24T06:18:50.036447+0000","last_fullsized":"2023-11-24T06:18:50.036447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023796+0000","last_change":"2023-11-24T06:18:50.025458+0000","last_active":"2023-11-24T06:18:51.023796+0000","last_peered":"2023-11-24T06:18:51.023796+0000","last_clean":"2023-11-24T06:18:51.023796+0000","last_became_active":"2023-11-24T06:18:50.025139+0000","last_became_peered":"2023-11-24T06:18:50.025139+0000","last_unstale":"2023-11-24T06:18:51.023796+0000","last_undegraded":"2023-11-24T06:18:51.023796+0000","last_fullsized":"2023-11-24T06:18:51.023796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:02:29.652618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.037010+0000","last_change":"2023-11-24T06:18:49.009105+0000","last_active":"2023-11-24T06:18:50.037010+0000","last_peered":"2023-11-24T06:18:50.037010+0000","last_clean":"2023-11-24T06:18:50.037010+0000","last_became_active":"2023-11-24T06:18:49.008802+0000","last_became_peered":"2023-11-24T06:18:49.008802+0000","last_unstale":"2023-11-24T06:18:50.037010+0000","last_undegraded":"2023-11-24T06:18:50.037010+0000","last_fullsized":"2023-11-24T06:18:50.037010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036544+0000","last_change":"2023-11-24T06:18:50.033700+0000","last_active":"2023-11-24T06:18:50.036544+0000","last_peered":"2023-11-24T06:18:50.036544+0000","last_clean":"2023-11-24T06:18:50.036544+0000","last_became_active":"2023-11-24T06:18:50.033372+0000","last_became_peered":"2023-11-24T06:18:50.033372+0000","last_unstale":"2023-11-24T06:18:50.036544+0000","last_undegraded":"2023-11-24T06:18:50.036544+0000","last_fullsized":"2023-11-24T06:18:50.036544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036391+0000","last_change":"2023-11-24T06:18:49.015649+0000","last_active":"2023-11-24T06:18:50.036391+0000","last_peered":"2023-11-24T06:18:50.036391+0000","last_clean":"2023-11-24T06:18:50.036391+0000","last_became_active":"2023-11-24T06:18:49.015329+0000","last_became_peered":"2023-11-24T06:18:49.015329+0000","last_unstale":"2023-11-24T06:18:50.036391+0000","last_undegraded":"2023-11-24T06:18:50.036391+0000","last_fullsized":"2023-11-24T06:18:50.036391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038969+0000","last_change":"2023-11-24T06:18:50.037859+0000","last_active":"2023-11-24T06:18:50.038969+0000","last_peered":"2023-11-24T06:18:50.038969+0000","last_clean":"2023-11-24T06:18:50.038969+0000","last_became_active":"2023-11-24T06:18:50.037565+0000","last_became_peered":"2023-11-24T06:18:50.037565+0000","last_unstale":"2023-11-24T06:18:50.038969+0000","last_undegraded":"2023-11-24T06:18:50.038969+0000","last_fullsized":"2023-11-24T06:18:50.038969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024207+0000","last_change":"2023-11-24T06:18:49.008317+0000","last_active":"2023-11-24T06:18:51.024207+0000","last_peered":"2023-11-24T06:18:51.024207+0000","last_clean":"2023-11-24T06:18:51.024207+0000","last_became_active":"2023-11-24T06:18:49.008039+0000","last_became_peered":"2023-11-24T06:18:49.008039+0000","last_unstale":"2023-11-24T06:18:51.024207+0000","last_undegraded":"2023-11-24T06:18:51.024207+0000","last_fullsized":"2023-11-24T06:18:51.024207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:58.320250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.039648+0000","last_change":"2023-11-24T06:18:50.034882+0000","last_active":"2023-11-24T06:18:50.039648+0000","last_peered":"2023-11-24T06:18:50.039648+0000","last_clean":"2023-11-24T06:18:50.039648+0000","last_became_active":"2023-11-24T06:18:50.031623+0000","last_became_peered":"2023-11-24T06:18:50.031623+0000","last_unstale":"2023-11-24T06:18:50.039648+0000","last_undegraded":"2023-11-24T06:18:50.039648+0000","last_fullsized":"2023-11-24T06:18:50.039648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:54.282518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023720+0000","last_change":"2023-11-24T06:18:49.007346+0000","last_active":"2023-11-24T06:18:51.023720+0000","last_peered":"2023-11-24T06:18:51.023720+0000","last_clean":"2023-11-24T06:18:51.023720+0000","last_became_active":"2023-11-24T06:18:49.006300+0000","last_became_peered":"2023-11-24T06:18:49.006300+0000","last_unstale":"2023-11-24T06:18:51.023720+0000","last_undegraded":"2023-11-24T06:18:51.023720+0000","last_fullsized":"2023-11-24T06:18:51.023720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.037998+0000","last_change":"2023-11-24T06:18:50.029883+0000","last_active":"2023-11-24T06:18:50.037998+0000","last_peered":"2023-11-24T06:18:50.037998+0000","last_clean":"2023-11-24T06:18:50.037998+0000","last_became_active":"2023-11-24T06:18:50.029618+0000","last_became_peered":"2023-11-24T06:18:50.029618+0000","last_unstale":"2023-11-24T06:18:50.037998+0000","last_undegraded":"2023-11-24T06:18:50.037998+0000","last_fullsized":"2023-11-24T06:18:50.037998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036344+0000","last_change":"2023-11-24T06:18:49.015809+0000","last_active":"2023-11-24T06:18:50.036344+0000","last_peered":"2023-11-24T06:18:50.036344+0000","last_clean":"2023-11-24T06:18:50.036344+0000","last_became_active":"2023-11-24T06:18:49.015073+0000","last_became_peered":"2023-11-24T06:18:49.015073+0000","last_unstale":"2023-11-24T06:18:50.036344+0000","last_undegraded":"2023-11-24T06:18:50.036344+0000","last_fullsized":"2023-11-24T06:18:50.036344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:40.335012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023148+0000","last_change":"2023-11-24T06:18:50.015301+0000","last_active":"2023-11-24T06:18:51.023148+0000","last_peered":"2023-11-24T06:18:51.023148+0000","last_clean":"2023-11-24T06:18:51.023148+0000","last_became_active":"2023-11-24T06:18:50.015007+0000","last_became_peered":"2023-11-24T06:18:50.015007+0000","last_unstale":"2023-11-24T06:18:51.023148+0000","last_undegraded":"2023-11-24T06:18:51.023148+0000","last_fullsized":"2023-11-24T06:18:51.023148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.033470+0000","last_change":"2023-11-24T06:18:49.008047+0000","last_active":"2023-11-24T06:18:50.033470+0000","last_peered":"2023-11-24T06:18:50.033470+0000","last_clean":"2023-11-24T06:18:50.033470+0000","last_became_active":"2023-11-24T06:18:49.007764+0000","last_became_peered":"2023-11-24T06:18:49.007764+0000","last_unstale":"2023-11-24T06:18:50.033470+0000","last_undegraded":"2023-11-24T06:18:50.033470+0000","last_fullsized":"2023-11-24T06:18:50.033470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:02.607697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032842+0000","last_change":"2023-11-24T06:18:50.027945+0000","last_active":"2023-11-24T06:18:50.032842+0000","last_peered":"2023-11-24T06:18:50.032842+0000","last_clean":"2023-11-24T06:18:50.032842+0000","last_became_active":"2023-11-24T06:18:50.027609+0000","last_became_peered":"2023-11-24T06:18:50.027609+0000","last_unstale":"2023-11-24T06:18:50.032842+0000","last_undegraded":"2023-11-24T06:18:50.032842+0000","last_fullsized":"2023-11-24T06:18:50.032842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:52.017798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.019687+0000","last_change":"2023-11-24T06:18:49.014611+0000","last_active":"2023-11-24T06:18:50.019687+0000","last_peered":"2023-11-24T06:18:50.019687+0000","last_clean":"2023-11-24T06:18:50.019687+0000","last_became_active":"2023-11-24T06:18:49.014277+0000","last_became_peered":"2023-11-24T06:18:49.014277+0000","last_unstale":"2023-11-24T06:18:50.019687+0000","last_undegraded":"2023-11-24T06:18:50.019687+0000","last_fullsized":"2023-11-24T06:18:50.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:33.478815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035905+0000","last_change":"2023-11-24T06:18:50.027283+0000","last_active":"2023-11-24T06:18:50.035905+0000","last_peered":"2023-11-24T06:18:50.035905+0000","last_clean":"2023-11-24T06:18:50.035905+0000","last_became_active":"2023-11-24T06:18:50.027028+0000","last_became_peered":"2023-11-24T06:18:50.027028+0000","last_unstale":"2023-11-24T06:18:50.035905+0000","last_undegraded":"2023-11-24T06:18:50.035905+0000","last_fullsized":"2023-11-24T06:18:50.035905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:24.709998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.019779+0000","last_change":"2023-11-24T06:18:49.007327+0000","last_active":"2023-11-24T06:18:50.019779+0000","last_peered":"2023-11-24T06:18:50.019779+0000","last_clean":"2023-11-24T06:18:50.019779+0000","last_became_active":"2023-11-24T06:18:49.006684+0000","last_became_peered":"2023-11-24T06:18:49.006684+0000","last_unstale":"2023-11-24T06:18:50.019779+0000","last_undegraded":"2023-11-24T06:18:50.019779+0000","last_fullsized":"2023-11-24T06:18:50.019779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:54.335883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021189+0000","last_change":"2023-11-24T06:18:50.026294+0000","last_active":"2023-11-24T06:18:51.021189+0000","last_peered":"2023-11-24T06:18:51.021189+0000","last_clean":"2023-11-24T06:18:51.021189+0000","last_became_active":"2023-11-24T06:18:50.026020+0000","last_became_peered":"2023-11-24T06:18:50.026020+0000","last_unstale":"2023-11-24T06:18:51.021189+0000","last_undegraded":"2023-11-24T06:18:51.021189+0000","last_fullsized":"2023-11-24T06:18:51.021189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.017768+0000","last_change":"2023-11-24T06:18:49.011149+0000","last_active":"2023-11-24T06:18:50.017768+0000","last_peered":"2023-11-24T06:18:50.017768+0000","last_clean":"2023-11-24T06:18:50.017768+0000","last_became_active":"2023-11-24T06:18:49.010573+0000","last_became_peered":"2023-11-24T06:18:49.010573+0000","last_unstale":"2023-11-24T06:18:50.017768+0000","last_undegraded":"2023-11-24T06:18:50.017768+0000","last_fullsized":"2023-11-24T06:18:50.017768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040771+0000","last_change":"2023-11-24T06:18:50.035496+0000","last_active":"2023-11-24T06:18:50.040771+0000","last_peered":"2023-11-24T06:18:50.040771+0000","last_clean":"2023-11-24T06:18:50.040771+0000","last_became_active":"2023-11-24T06:18:50.035172+0000","last_became_peered":"2023-11-24T06:18:50.035172+0000","last_unstale":"2023-11-24T06:18:50.040771+0000","last_undegraded":"2023-11-24T06:18:50.040771+0000","last_fullsized":"2023-11-24T06:18:50.040771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:56:59.481984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.014777+0000","last_change":"2023-11-24T06:18:49.012031+0000","last_active":"2023-11-24T06:18:50.014777+0000","last_peered":"2023-11-24T06:18:50.014777+0000","last_clean":"2023-11-24T06:18:50.014777+0000","last_became_active":"2023-11-24T06:18:49.011741+0000","last_became_peered":"2023-11-24T06:18:49.011741+0000","last_unstale":"2023-11-24T06:18:50.014777+0000","last_undegraded":"2023-11-24T06:18:50.014777+0000","last_fullsized":"2023-11-24T06:18:50.014777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038700+0000","last_change":"2023-11-24T06:18:50.035543+0000","last_active":"2023-11-24T06:18:50.038700+0000","last_peered":"2023-11-24T06:18:50.038700+0000","last_clean":"2023-11-24T06:18:50.038700+0000","last_became_active":"2023-11-24T06:18:50.035269+0000","last_became_peered":"2023-11-24T06:18:50.035269+0000","last_unstale":"2023-11-24T06:18:50.038700+0000","last_undegraded":"2023-11-24T06:18:50.038700+0000","last_fullsized":"2023-11-24T06:18:50.038700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021480+0000","last_change":"2023-11-24T06:18:49.006503+0000","last_active":"2023-11-24T06:18:51.021480+0000","last_peered":"2023-11-24T06:18:51.021480+0000","last_clean":"2023-11-24T06:18:51.021480+0000","last_became_active":"2023-11-24T06:18:49.006176+0000","last_became_peered":"2023-11-24T06:18:49.006176+0000","last_unstale":"2023-11-24T06:18:51.021480+0000","last_undegraded":"2023-11-24T06:18:51.021480+0000","last_fullsized":"2023-11-24T06:18:51.021480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:33.478815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023461+0000","last_change":"2023-11-24T06:18:50.023420+0000","last_active":"2023-11-24T06:18:51.023461+0000","last_peered":"2023-11-24T06:18:51.023461+0000","last_clean":"2023-11-24T06:18:51.023461+0000","last_became_active":"2023-11-24T06:18:50.022722+0000","last_became_peered":"2023-11-24T06:18:50.022722+0000","last_unstale":"2023-11-24T06:18:51.023461+0000","last_undegraded":"2023-11-24T06:18:51.023461+0000","last_fullsized":"2023-11-24T06:18:51.023461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:24.709998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040671+0000","last_change":"2023-11-24T06:18:49.016525+0000","last_active":"2023-11-24T06:18:50.040671+0000","last_peered":"2023-11-24T06:18:50.040671+0000","last_clean":"2023-11-24T06:18:50.040671+0000","last_became_active":"2023-11-24T06:18:49.016259+0000","last_became_peered":"2023-11-24T06:18:49.016259+0000","last_unstale":"2023-11-24T06:18:50.040671+0000","last_undegraded":"2023-11-24T06:18:50.040671+0000","last_fullsized":"2023-11-24T06:18:50.040671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:13.224297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023938+0000","last_change":"2023-11-24T06:18:50.024377+0000","last_active":"2023-11-24T06:18:51.023938+0000","last_peered":"2023-11-24T06:18:51.023938+0000","last_clean":"2023-11-24T06:18:51.023938+0000","last_became_active":"2023-11-24T06:18:50.024123+0000","last_became_peered":"2023-11-24T06:18:50.024123+0000","last_unstale":"2023-11-24T06:18:51.023938+0000","last_undegraded":"2023-11-24T06:18:51.023938+0000","last_fullsized":"2023-11-24T06:18:51.023938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:28:39.352489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041197+0000","last_change":"2023-11-24T06:18:49.009176+0000","last_active":"2023-11-24T06:18:50.041197+0000","last_peered":"2023-11-24T06:18:50.041197+0000","last_clean":"2023-11-24T06:18:50.041197+0000","last_became_active":"2023-11-24T06:18:49.008875+0000","last_became_peered":"2023-11-24T06:18:49.008875+0000","last_unstale":"2023-11-24T06:18:50.041197+0000","last_undegraded":"2023-11-24T06:18:50.041197+0000","last_fullsized":"2023-11-24T06:18:50.041197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.034300+0000","last_change":"2023-11-24T06:18:50.032393+0000","last_active":"2023-11-24T06:18:50.034300+0000","last_peered":"2023-11-24T06:18:50.034300+0000","last_clean":"2023-11-24T06:18:50.034300+0000","last_became_active":"2023-11-24T06:18:50.032145+0000","last_became_peered":"2023-11-24T06:18:50.032145+0000","last_unstale":"2023-11-24T06:18:50.034300+0000","last_undegraded":"2023-11-24T06:18:50.034300+0000","last_fullsized":"2023-11-24T06:18:50.034300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.020491+0000","last_change":"2023-11-24T06:18:49.015579+0000","last_active":"2023-11-24T06:18:50.020491+0000","last_peered":"2023-11-24T06:18:50.020491+0000","last_clean":"2023-11-24T06:18:50.020491+0000","last_became_active":"2023-11-24T06:18:49.015329+0000","last_became_peered":"2023-11-24T06:18:49.015329+0000","last_unstale":"2023-11-24T06:18:50.020491+0000","last_undegraded":"2023-11-24T06:18:50.020491+0000","last_fullsized":"2023-11-24T06:18:50.020491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:37.250753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024128+0000","last_change":"2023-11-24T06:18:50.021964+0000","last_active":"2023-11-24T06:18:51.024128+0000","last_peered":"2023-11-24T06:18:51.024128+0000","last_clean":"2023-11-24T06:18:51.024128+0000","last_became_active":"2023-11-24T06:18:50.021678+0000","last_became_peered":"2023-11-24T06:18:50.021678+0000","last_unstale":"2023-11-24T06:18:51.024128+0000","last_undegraded":"2023-11-24T06:18:51.024128+0000","last_fullsized":"2023-11-24T06:18:51.024128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:32:20.935789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.017027+0000","last_change":"2023-11-24T06:18:49.007859+0000","last_active":"2023-11-24T06:18:50.017027+0000","last_peered":"2023-11-24T06:18:50.017027+0000","last_clean":"2023-11-24T06:18:50.017027+0000","last_became_active":"2023-11-24T06:18:49.007569+0000","last_became_peered":"2023-11-24T06:18:49.007569+0000","last_unstale":"2023-11-24T06:18:50.017027+0000","last_undegraded":"2023-11-24T06:18:50.017027+0000","last_fullsized":"2023-11-24T06:18:50.017027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038984+0000","last_change":"2023-11-24T06:18:50.033953+0000","last_active":"2023-11-24T06:18:50.038984+0000","last_peered":"2023-11-24T06:18:50.038984+0000","last_clean":"2023-11-24T06:18:50.038984+0000","last_became_active":"2023-11-24T06:18:50.033664+0000","last_became_peered":"2023-11-24T06:18:50.033664+0000","last_unstale":"2023-11-24T06:18:50.038984+0000","last_undegraded":"2023-11-24T06:18:50.038984+0000","last_fullsized":"2023-11-24T06:18:50.038984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:54.282518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.037164+0000","last_change":"2023-11-24T06:18:49.009433+0000","last_active":"2023-11-24T06:18:50.037164+0000","last_peered":"2023-11-24T06:18:50.037164+0000","last_clean":"2023-11-24T06:18:50.037164+0000","last_became_active":"2023-11-24T06:18:49.009133+0000","last_became_peered":"2023-11-24T06:18:49.009133+0000","last_unstale":"2023-11-24T06:18:50.037164+0000","last_undegraded":"2023-11-24T06:18:50.037164+0000","last_fullsized":"2023-11-24T06:18:50.037164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023055+0000","last_change":"2023-11-24T06:18:50.023048+0000","last_active":"2023-11-24T06:18:51.023055+0000","last_peered":"2023-11-24T06:18:51.023055+0000","last_clean":"2023-11-24T06:18:51.023055+0000","last_became_active":"2023-11-24T06:18:50.022736+0000","last_became_peered":"2023-11-24T06:18:50.022736+0000","last_unstale":"2023-11-24T06:18:51.023055+0000","last_undegraded":"2023-11-24T06:18:51.023055+0000","last_fullsized":"2023-11-24T06:18:51.023055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:14.887594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023699+0000","last_change":"2023-11-24T06:18:49.012343+0000","last_active":"2023-11-24T06:18:51.023699+0000","last_peered":"2023-11-24T06:18:51.023699+0000","last_clean":"2023-11-24T06:18:51.023699+0000","last_became_active":"2023-11-24T06:18:49.012078+0000","last_became_peered":"2023-11-24T06:18:49.012078+0000","last_unstale":"2023-11-24T06:18:51.023699+0000","last_undegraded":"2023-11-24T06:18:51.023699+0000","last_fullsized":"2023-11-24T06:18:51.023699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024206+0000","last_change":"2023-11-24T06:18:50.022294+0000","last_active":"2023-11-24T06:18:51.024206+0000","last_peered":"2023-11-24T06:18:51.024206+0000","last_clean":"2023-11-24T06:18:51.024206+0000","last_became_active":"2023-11-24T06:18:50.021806+0000","last_became_peered":"2023-11-24T06:18:50.021806+0000","last_unstale":"2023-11-24T06:18:51.024206+0000","last_undegraded":"2023-11-24T06:18:51.024206+0000","last_fullsized":"2023-11-24T06:18:51.024206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:03.577619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021419+0000","last_change":"2023-11-24T06:18:49.010001+0000","last_active":"2023-11-24T06:18:51.021419+0000","last_peered":"2023-11-24T06:18:51.021419+0000","last_clean":"2023-11-24T06:18:51.021419+0000","last_became_active":"2023-11-24T06:18:49.009729+0000","last_became_peered":"2023-11-24T06:18:49.009729+0000","last_unstale":"2023-11-24T06:18:51.021419+0000","last_undegraded":"2023-11-24T06:18:51.021419+0000","last_fullsized":"2023-11-24T06:18:51.021419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:24.571482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040814+0000","last_change":"2023-11-24T06:18:50.035809+0000","last_active":"2023-11-24T06:18:50.040814+0000","last_peered":"2023-11-24T06:18:50.040814+0000","last_clean":"2023-11-24T06:18:50.040814+0000","last_became_active":"2023-11-24T06:18:50.033211+0000","last_became_peered":"2023-11-24T06:18:50.033211+0000","last_unstale":"2023-11-24T06:18:50.040814+0000","last_undegraded":"2023-11-24T06:18:50.040814+0000","last_fullsized":"2023-11-24T06:18:50.040814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.023048+0000","last_change":"2023-11-24T06:18:49.004923+0000","last_active":"2023-11-24T06:18:50.023048+0000","last_peered":"2023-11-24T06:18:50.023048+0000","last_clean":"2023-11-24T06:18:50.023048+0000","last_became_active":"2023-11-24T06:18:49.004601+0000","last_became_peered":"2023-11-24T06:18:49.004601+0000","last_unstale":"2023-11-24T06:18:50.023048+0000","last_undegraded":"2023-11-24T06:18:50.023048+0000","last_fullsized":"2023-11-24T06:18:50.023048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:42.514481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023365+0000","last_change":"2023-11-24T06:18:50.026574+0000","last_active":"2023-11-24T06:18:51.023365+0000","last_peered":"2023-11-24T06:18:51.023365+0000","last_clean":"2023-11-24T06:18:51.023365+0000","last_became_active":"2023-11-24T06:18:50.026028+0000","last_became_peered":"2023-11-24T06:18:50.026028+0000","last_unstale":"2023-11-24T06:18:51.023365+0000","last_undegraded":"2023-11-24T06:18:51.023365+0000","last_fullsized":"2023-11-24T06:18:51.023365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021109+0000","last_change":"2023-11-24T06:18:50.027544+0000","last_active":"2023-11-24T06:18:51.021109+0000","last_peered":"2023-11-24T06:18:51.021109+0000","last_clean":"2023-11-24T06:18:51.021109+0000","last_became_active":"2023-11-24T06:18:50.027272+0000","last_became_peered":"2023-11-24T06:18:50.027272+0000","last_unstale":"2023-11-24T06:18:51.021109+0000","last_undegraded":"2023-11-24T06:18:51.021109+0000","last_fullsized":"2023-11-24T06:18:51.021109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.019067+0000","last_change":"2023-11-24T06:18:49.015756+0000","last_active":"2023-11-24T06:18:50.019067+0000","last_peered":"2023-11-24T06:18:50.019067+0000","last_clean":"2023-11-24T06:18:50.019067+0000","last_became_active":"2023-11-24T06:18:49.015446+0000","last_became_peered":"2023-11-24T06:18:49.015446+0000","last_unstale":"2023-11-24T06:18:50.019067+0000","last_undegraded":"2023-11-24T06:18:50.019067+0000","last_fullsized":"2023-11-24T06:18:50.019067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:24.571482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040735+0000","last_change":"2023-11-24T06:18:50.040018+0000","last_active":"2023-11-24T06:18:50.040735+0000","last_peered":"2023-11-24T06:18:50.040735+0000","last_clean":"2023-11-24T06:18:50.040735+0000","last_became_active":"2023-11-24T06:18:50.039786+0000","last_became_peered":"2023-11-24T06:18:50.039786+0000","last_unstale":"2023-11-24T06:18:50.040735+0000","last_undegraded":"2023-11-24T06:18:50.040735+0000","last_fullsized":"2023-11-24T06:18:50.040735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:27.849578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.014883+0000","last_change":"2023-11-24T06:18:49.009457+0000","last_active":"2023-11-24T06:18:50.014883+0000","last_peered":"2023-11-24T06:18:50.014883+0000","last_clean":"2023-11-24T06:18:50.014883+0000","last_became_active":"2023-11-24T06:18:49.008932+0000","last_became_peered":"2023-11-24T06:18:49.008932+0000","last_unstale":"2023-11-24T06:18:50.014883+0000","last_undegraded":"2023-11-24T06:18:50.014883+0000","last_fullsized":"2023-11-24T06:18:50.014883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.009997+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:18:50.009997+0000","last_peered":"2023-11-24T06:18:50.009997+0000","last_clean":"2023-11-24T06:18:50.009997+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:18:50.009997+0000","last_undegraded":"2023-11-24T06:18:50.009997+0000","last_fullsized":"2023-11-24T06:18:50.009997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032512+0000","last_change":"2023-11-24T06:18:50.027632+0000","last_active":"2023-11-24T06:18:50.032512+0000","last_peered":"2023-11-24T06:18:50.032512+0000","last_clean":"2023-11-24T06:18:50.032512+0000","last_became_active":"2023-11-24T06:18:50.027344+0000","last_became_peered":"2023-11-24T06:18:50.027344+0000","last_unstale":"2023-11-24T06:18:50.032512+0000","last_undegraded":"2023-11-24T06:18:50.032512+0000","last_fullsized":"2023-11-24T06:18:50.032512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:25.578418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022810+0000","last_change":"2023-11-24T06:18:49.006996+0000","last_active":"2023-11-24T06:18:51.022810+0000","last_peered":"2023-11-24T06:18:51.022810+0000","last_clean":"2023-11-24T06:18:51.022810+0000","last_became_active":"2023-11-24T06:18:49.006652+0000","last_became_peered":"2023-11-24T06:18:49.006652+0000","last_unstale":"2023-11-24T06:18:51.022810+0000","last_undegraded":"2023-11-24T06:18:51.022810+0000","last_fullsized":"2023-11-24T06:18:51.022810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:01.167887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024055+0000","last_change":"2023-11-24T06:18:49.012066+0000","last_active":"2023-11-24T06:18:51.024055+0000","last_peered":"2023-11-24T06:18:51.024055+0000","last_clean":"2023-11-24T06:18:51.024055+0000","last_became_active":"2023-11-24T06:18:49.011795+0000","last_became_peered":"2023-11-24T06:18:49.011795+0000","last_unstale":"2023-11-24T06:18:51.024055+0000","last_undegraded":"2023-11-24T06:18:51.024055+0000","last_fullsized":"2023-11-24T06:18:51.024055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:22.850843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021314+0000","last_change":"2023-11-24T06:18:50.021739+0000","last_active":"2023-11-24T06:18:51.021314+0000","last_peered":"2023-11-24T06:18:51.021314+0000","last_clean":"2023-11-24T06:18:51.021314+0000","last_became_active":"2023-11-24T06:18:50.021452+0000","last_became_peered":"2023-11-24T06:18:50.021452+0000","last_unstale":"2023-11-24T06:18:51.021314+0000","last_undegraded":"2023-11-24T06:18:51.021314+0000","last_fullsized":"2023-11-24T06:18:51.021314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021363+0000","last_change":"2023-11-24T06:18:49.012066+0000","last_active":"2023-11-24T06:18:51.021363+0000","last_peered":"2023-11-24T06:18:51.021363+0000","last_clean":"2023-11-24T06:18:51.021363+0000","last_became_active":"2023-11-24T06:18:49.011748+0000","last_became_peered":"2023-11-24T06:18:49.011748+0000","last_unstale":"2023-11-24T06:18:51.021363+0000","last_undegraded":"2023-11-24T06:18:51.021363+0000","last_fullsized":"2023-11-24T06:18:51.021363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036495+0000","last_change":"2023-11-24T06:18:50.033206+0000","last_active":"2023-11-24T06:18:50.036495+0000","last_peered":"2023-11-24T06:18:50.036495+0000","last_clean":"2023-11-24T06:18:50.036495+0000","last_became_active":"2023-11-24T06:18:50.032878+0000","last_became_peered":"2023-11-24T06:18:50.032878+0000","last_unstale":"2023-11-24T06:18:50.036495+0000","last_undegraded":"2023-11-24T06:18:50.036495+0000","last_fullsized":"2023-11-24T06:18:50.036495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:03.614632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036352+0000","last_change":"2023-11-24T06:18:49.011356+0000","last_active":"2023-11-24T06:18:50.036352+0000","last_peered":"2023-11-24T06:18:50.036352+0000","last_clean":"2023-11-24T06:18:50.036352+0000","last_became_active":"2023-11-24T06:18:49.010903+0000","last_became_peered":"2023-11-24T06:18:49.010903+0000","last_unstale":"2023-11-24T06:18:50.036352+0000","last_undegraded":"2023-11-24T06:18:50.036352+0000","last_fullsized":"2023-11-24T06:18:50.036352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032088+0000","last_change":"2023-11-24T06:18:50.024110+0000","last_active":"2023-11-24T06:18:50.032088+0000","last_peered":"2023-11-24T06:18:50.032088+0000","last_clean":"2023-11-24T06:18:50.032088+0000","last_became_active":"2023-11-24T06:18:50.023801+0000","last_became_peered":"2023-11-24T06:18:50.023801+0000","last_unstale":"2023-11-24T06:18:50.032088+0000","last_undegraded":"2023-11-24T06:18:50.032088+0000","last_fullsized":"2023-11-24T06:18:50.032088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:03.614632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022707+0000","last_change":"2023-11-24T06:18:49.010428+0000","last_active":"2023-11-24T06:18:51.022707+0000","last_peered":"2023-11-24T06:18:51.022707+0000","last_clean":"2023-11-24T06:18:51.022707+0000","last_became_active":"2023-11-24T06:18:49.010124+0000","last_became_peered":"2023-11-24T06:18:49.010124+0000","last_unstale":"2023-11-24T06:18:51.022707+0000","last_undegraded":"2023-11-24T06:18:51.022707+0000","last_fullsized":"2023-11-24T06:18:51.022707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023378+0000","last_change":"2023-11-24T06:18:50.022092+0000","last_active":"2023-11-24T06:18:51.023378+0000","last_peered":"2023-11-24T06:18:51.023378+0000","last_clean":"2023-11-24T06:18:51.023378+0000","last_became_active":"2023-11-24T06:18:50.021808+0000","last_became_peered":"2023-11-24T06:18:50.021808+0000","last_unstale":"2023-11-24T06:18:51.023378+0000","last_undegraded":"2023-11-24T06:18:51.023378+0000","last_fullsized":"2023-11-24T06:18:51.023378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:21.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021058+0000","last_change":"2023-11-24T06:18:49.007474+0000","last_active":"2023-11-24T06:18:51.021058+0000","last_peered":"2023-11-24T06:18:51.021058+0000","last_clean":"2023-11-24T06:18:51.021058+0000","last_became_active":"2023-11-24T06:18:49.007228+0000","last_became_peered":"2023-11-24T06:18:49.007228+0000","last_unstale":"2023-11-24T06:18:51.021058+0000","last_undegraded":"2023-11-24T06:18:51.021058+0000","last_fullsized":"2023-11-24T06:18:51.021058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035548+0000","last_change":"2023-11-24T06:18:50.025141+0000","last_active":"2023-11-24T06:18:50.035548+0000","last_peered":"2023-11-24T06:18:50.035548+0000","last_clean":"2023-11-24T06:18:50.035548+0000","last_became_active":"2023-11-24T06:18:50.024859+0000","last_became_peered":"2023-11-24T06:18:50.024859+0000","last_unstale":"2023-11-24T06:18:50.035548+0000","last_undegraded":"2023-11-24T06:18:50.035548+0000","last_fullsized":"2023-11-24T06:18:50.035548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021233+0000","last_change":"2023-11-24T06:18:49.015615+0000","last_active":"2023-11-24T06:18:51.021233+0000","last_peered":"2023-11-24T06:18:51.021233+0000","last_clean":"2023-11-24T06:18:51.021233+0000","last_became_active":"2023-11-24T06:18:49.015320+0000","last_became_peered":"2023-11-24T06:18:49.015320+0000","last_unstale":"2023-11-24T06:18:51.021233+0000","last_undegraded":"2023-11-24T06:18:51.021233+0000","last_fullsized":"2023-11-24T06:18:51.021233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035617+0000","last_change":"2023-11-24T06:18:50.025429+0000","last_active":"2023-11-24T06:18:50.035617+0000","last_peered":"2023-11-24T06:18:50.035617+0000","last_clean":"2023-11-24T06:18:50.035617+0000","last_became_active":"2023-11-24T06:18:50.024972+0000","last_became_peered":"2023-11-24T06:18:50.024972+0000","last_unstale":"2023-11-24T06:18:50.035617+0000","last_undegraded":"2023-11-24T06:18:50.035617+0000","last_fullsized":"2023-11-24T06:18:50.035617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:47:17.509901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024261+0000","last_change":"2023-11-24T06:18:49.007100+0000","last_active":"2023-11-24T06:18:51.024261+0000","last_peered":"2023-11-24T06:18:51.024261+0000","last_clean":"2023-11-24T06:18:51.024261+0000","last_became_active":"2023-11-24T06:18:49.006577+0000","last_became_peered":"2023-11-24T06:18:49.006577+0000","last_unstale":"2023-11-24T06:18:51.024261+0000","last_undegraded":"2023-11-24T06:18:51.024261+0000","last_fullsized":"2023-11-24T06:18:51.024261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.033566+0000","last_change":"2023-11-24T06:18:50.032437+0000","last_active":"2023-11-24T06:18:50.033566+0000","last_peered":"2023-11-24T06:18:50.033566+0000","last_clean":"2023-11-24T06:18:50.033566+0000","last_became_active":"2023-11-24T06:18:50.032082+0000","last_became_peered":"2023-11-24T06:18:50.032082+0000","last_unstale":"2023-11-24T06:18:50.033566+0000","last_undegraded":"2023-11-24T06:18:50.033566+0000","last_fullsized":"2023-11-24T06:18:50.033566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.019270+0000","last_change":"2023-11-24T06:18:49.011290+0000","last_active":"2023-11-24T06:18:50.019270+0000","last_peered":"2023-11-24T06:18:50.019270+0000","last_clean":"2023-11-24T06:18:50.019270+0000","last_became_active":"2023-11-24T06:18:49.010889+0000","last_became_peered":"2023-11-24T06:18:49.010889+0000","last_unstale":"2023-11-24T06:18:50.019270+0000","last_undegraded":"2023-11-24T06:18:50.019270+0000","last_fullsized":"2023-11-24T06:18:50.019270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024098+0000","last_change":"2023-11-24T06:18:50.036167+0000","last_active":"2023-11-24T06:18:51.024098+0000","last_peered":"2023-11-24T06:18:51.024098+0000","last_clean":"2023-11-24T06:18:51.024098+0000","last_became_active":"2023-11-24T06:18:50.035904+0000","last_became_peered":"2023-11-24T06:18:50.035904+0000","last_unstale":"2023-11-24T06:18:51.024098+0000","last_undegraded":"2023-11-24T06:18:51.024098+0000","last_fullsized":"2023-11-24T06:18:51.024098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:33:00.547188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021448+0000","last_change":"2023-11-24T06:18:49.006925+0000","last_active":"2023-11-24T06:18:51.021448+0000","last_peered":"2023-11-24T06:18:51.021448+0000","last_clean":"2023-11-24T06:18:51.021448+0000","last_became_active":"2023-11-24T06:18:49.006482+0000","last_became_peered":"2023-11-24T06:18:49.006482+0000","last_unstale":"2023-11-24T06:18:51.021448+0000","last_undegraded":"2023-11-24T06:18:51.021448+0000","last_fullsized":"2023-11-24T06:18:51.021448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:40.335012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041428+0000","last_change":"2023-11-24T06:18:50.039417+0000","last_active":"2023-11-24T06:18:50.041428+0000","last_peered":"2023-11-24T06:18:50.041428+0000","last_clean":"2023-11-24T06:18:50.041428+0000","last_became_active":"2023-11-24T06:18:50.031461+0000","last_became_peered":"2023-11-24T06:18:50.031461+0000","last_unstale":"2023-11-24T06:18:50.041428+0000","last_undegraded":"2023-11-24T06:18:50.041428+0000","last_fullsized":"2023-11-24T06:18:50.041428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036803+0000","last_change":"2023-11-24T06:18:49.015786+0000","last_active":"2023-11-24T06:18:50.036803+0000","last_peered":"2023-11-24T06:18:50.036803+0000","last_clean":"2023-11-24T06:18:50.036803+0000","last_became_active":"2023-11-24T06:18:49.015506+0000","last_became_peered":"2023-11-24T06:18:49.015506+0000","last_unstale":"2023-11-24T06:18:50.036803+0000","last_undegraded":"2023-11-24T06:18:50.036803+0000","last_fullsized":"2023-11-24T06:18:50.036803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:01.167887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024358+0000","last_change":"2023-11-24T06:18:50.023402+0000","last_active":"2023-11-24T06:18:51.024358+0000","last_peered":"2023-11-24T06:18:51.024358+0000","last_clean":"2023-11-24T06:18:51.024358+0000","last_became_active":"2023-11-24T06:18:50.023117+0000","last_became_peered":"2023-11-24T06:18:50.023117+0000","last_unstale":"2023-11-24T06:18:51.024358+0000","last_undegraded":"2023-11-24T06:18:51.024358+0000","last_fullsized":"2023-11-24T06:18:51.024358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:54.022789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.034041+0000","last_change":"2023-11-24T06:18:50.032212+0000","last_active":"2023-11-24T06:18:50.034041+0000","last_peered":"2023-11-24T06:18:50.034041+0000","last_clean":"2023-11-24T06:18:50.034041+0000","last_became_active":"2023-11-24T06:18:50.031933+0000","last_became_peered":"2023-11-24T06:18:50.031933+0000","last_unstale":"2023-11-24T06:18:50.034041+0000","last_undegraded":"2023-11-24T06:18:50.034041+0000","last_fullsized":"2023-11-24T06:18:50.034041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038309+0000","last_change":"2023-11-24T06:18:50.037725+0000","last_active":"2023-11-24T06:18:50.038309+0000","last_peered":"2023-11-24T06:18:50.038309+0000","last_clean":"2023-11-24T06:18:50.038309+0000","last_became_active":"2023-11-24T06:18:50.037471+0000","last_became_peered":"2023-11-24T06:18:50.037471+0000","last_unstale":"2023-11-24T06:18:50.038309+0000","last_undegraded":"2023-11-24T06:18:50.038309+0000","last_fullsized":"2023-11-24T06:18:50.038309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041406+0000","last_change":"2023-11-24T06:18:50.038973+0000","last_active":"2023-11-24T06:18:50.041406+0000","last_peered":"2023-11-24T06:18:50.041406+0000","last_clean":"2023-11-24T06:18:50.041406+0000","last_became_active":"2023-11-24T06:18:50.038345+0000","last_became_peered":"2023-11-24T06:18:50.038345+0000","last_unstale":"2023-11-24T06:18:50.041406+0000","last_undegraded":"2023-11-24T06:18:50.041406+0000","last_fullsized":"2023-11-24T06:18:50.041406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:25.578418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032822+0000","last_change":"2023-11-24T06:18:50.024376+0000","last_active":"2023-11-24T06:18:50.032822+0000","last_peered":"2023-11-24T06:18:50.032822+0000","last_clean":"2023-11-24T06:18:50.032822+0000","last_became_active":"2023-11-24T06:18:50.024118+0000","last_became_peered":"2023-11-24T06:18:50.024118+0000","last_unstale":"2023-11-24T06:18:50.032822+0000","last_undegraded":"2023-11-24T06:18:50.032822+0000","last_fullsized":"2023-11-24T06:18:50.032822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:47:17.509901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036418+0000","last_change":"2023-11-24T06:18:50.028334+0000","last_active":"2023-11-24T06:18:50.036418+0000","last_peered":"2023-11-24T06:18:50.036418+0000","last_clean":"2023-11-24T06:18:50.036418+0000","last_became_active":"2023-11-24T06:18:50.028091+0000","last_became_peered":"2023-11-24T06:18:50.028091+0000","last_unstale":"2023-11-24T06:18:50.036418+0000","last_undegraded":"2023-11-24T06:18:50.036418+0000","last_fullsized":"2023-11-24T06:18:50.036418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021380+0000","last_change":"2023-11-24T06:18:50.025563+0000","last_active":"2023-11-24T06:18:51.021380+0000","last_peered":"2023-11-24T06:18:51.021380+0000","last_clean":"2023-11-24T06:18:51.021380+0000","last_became_active":"2023-11-24T06:18:50.025276+0000","last_became_peered":"2023-11-24T06:18:50.025276+0000","last_unstale":"2023-11-24T06:18:51.021380+0000","last_undegraded":"2023-11-24T06:18:51.021380+0000","last_fullsized":"2023-11-24T06:18:51.021380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.034911+0000","last_change":"2023-11-24T06:18:50.032672+0000","last_active":"2023-11-24T06:18:50.034911+0000","last_peered":"2023-11-24T06:18:50.034911+0000","last_clean":"2023-11-24T06:18:50.034911+0000","last_became_active":"2023-11-24T06:18:50.032400+0000","last_became_peered":"2023-11-24T06:18:50.032400+0000","last_unstale":"2023-11-24T06:18:50.034911+0000","last_undegraded":"2023-11-24T06:18:50.034911+0000","last_fullsized":"2023-11-24T06:18:50.034911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035709+0000","last_change":"2023-11-24T06:18:50.027189+0000","last_active":"2023-11-24T06:18:50.035709+0000","last_peered":"2023-11-24T06:18:50.035709+0000","last_clean":"2023-11-24T06:18:50.035709+0000","last_became_active":"2023-11-24T06:18:50.026893+0000","last_became_peered":"2023-11-24T06:18:50.026893+0000","last_unstale":"2023-11-24T06:18:50.035709+0000","last_undegraded":"2023-11-24T06:18:50.035709+0000","last_fullsized":"2023-11-24T06:18:50.035709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:21.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035986+0000","last_change":"2023-11-24T06:18:50.031274+0000","last_active":"2023-11-24T06:18:50.035986+0000","last_peered":"2023-11-24T06:18:50.035986+0000","last_clean":"2023-11-24T06:18:50.035986+0000","last_became_active":"2023-11-24T06:18:50.030997+0000","last_became_peered":"2023-11-24T06:18:50.030997+0000","last_unstale":"2023-11-24T06:18:50.035986+0000","last_undegraded":"2023-11-24T06:18:50.035986+0000","last_fullsized":"2023-11-24T06:18:50.035986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035754+0000","last_change":"2023-11-24T06:18:50.027675+0000","last_active":"2023-11-24T06:18:50.035754+0000","last_peered":"2023-11-24T06:18:50.035754+0000","last_clean":"2023-11-24T06:18:50.035754+0000","last_became_active":"2023-11-24T06:18:50.027402+0000","last_became_peered":"2023-11-24T06:18:50.027402+0000","last_unstale":"2023-11-24T06:18:50.035754+0000","last_undegraded":"2023-11-24T06:18:50.035754+0000","last_fullsized":"2023-11-24T06:18:50.035754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041439+0000","last_change":"2023-11-24T06:18:50.039174+0000","last_active":"2023-11-24T06:18:50.041439+0000","last_peered":"2023-11-24T06:18:50.041439+0000","last_clean":"2023-11-24T06:18:50.041439+0000","last_became_active":"2023-11-24T06:18:50.038671+0000","last_became_peered":"2023-11-24T06:18:50.038671+0000","last_unstale":"2023-11-24T06:18:50.041439+0000","last_undegraded":"2023-11-24T06:18:50.041439+0000","last_fullsized":"2023-11-24T06:18:50.041439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:03.614632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.034395+0000","last_change":"2023-11-24T06:18:50.032515+0000","last_active":"2023-11-24T06:18:50.034395+0000","last_peered":"2023-11-24T06:18:50.034395+0000","last_clean":"2023-11-24T06:18:50.034395+0000","last_became_active":"2023-11-24T06:18:50.032238+0000","last_became_peered":"2023-11-24T06:18:50.032238+0000","last_unstale":"2023-11-24T06:18:50.034395+0000","last_undegraded":"2023-11-24T06:18:50.034395+0000","last_fullsized":"2023-11-24T06:18:50.034395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.039901+0000","last_change":"2023-11-24T06:18:50.035460+0000","last_active":"2023-11-24T06:18:50.039901+0000","last_peered":"2023-11-24T06:18:50.039901+0000","last_clean":"2023-11-24T06:18:50.039901+0000","last_became_active":"2023-11-24T06:18:50.031929+0000","last_became_peered":"2023-11-24T06:18:50.031929+0000","last_unstale":"2023-11-24T06:18:50.039901+0000","last_undegraded":"2023-11-24T06:18:50.039901+0000","last_fullsized":"2023-11-24T06:18:50.039901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":660287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9289999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1629999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":69371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:18:54.124 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:18:54.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.276+0000 7fa8d98fc700 1 Processor -- start 2023-11-24T06:18:54.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.276+0000 7fa8d98fc700 1 -- start start 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d98fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d40b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d98fc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa8d406c710 0x7fa8d40b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d98fc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d40af950 con 0x7fa8d40adf00 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d98fc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d4144a80 con 0x7fa8d406c710 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d98fc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40afc30 con 0x7fa8d41446a0 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d2ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d40b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d37fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa8d406c710 0x7fa8d40b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d2ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d40b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:43932/0 (socket says 172.21.15.89:43932) 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d37fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa8d406c710 0x7fa8d40b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50172/0 (socket says 172.21.15.89:50172) 2023-11-24T06:18:54.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d2ffd700 1 -- 172.21.15.89:0/3733835989 learned_addr learned my addr 172.21.15.89:0/3733835989 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d27fc700 1 --1- 172.21.15.89:0/3733835989 >> v1:172.21.15.123:6789/0 conn(0x7fa8d40adf00 0x7fa8d406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:42384/0 (socket says 172.21.15.89:42384) 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d37fe700 1 -- 172.21.15.89:0/3733835989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 msgr2=0x7fa8d40b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d37fe700 1 --2- 172.21.15.89:0/3733835989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d40b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d37fe700 1 -- 172.21.15.89:0/3733835989 >> v1:172.21.15.123:6789/0 conn(0x7fa8d40adf00 legacy=0x7fa8d406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d37fe700 1 -- 172.21.15.89:0/3733835989 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d40aff10 con 0x7fa8d406c710 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.278+0000 7fa8d2ffd700 1 --2- 172.21.15.89:0/3733835989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d40b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d37fe700 1 --2- 172.21.15.89:0/3733835989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 0x7fa8d40b3d40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009d90 tx=0x7fa8cc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0b4fb17b0d2aca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d1ffb700 1 -- 172.21.15.89:0/3733835989 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc00e070 con 0x7fa8d406c710 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d1ffb700 1 -- 172.21.15.89:0/3733835989 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc00bb20 con 0x7fa8d406c710 2023-11-24T06:18:54.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d1ffb700 1 -- 172.21.15.89:0/3733835989 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc012430 con 0x7fa8d406c710 2023-11-24T06:18:54.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3733835989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 msgr2=0x7fa8d40b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3733835989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 0x7fa8d40b3d40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009d90 tx=0x7fa8cc00b330 comp rx=0 tx=0).stop 2023-11-24T06:18:54.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3733835989 shutdown_connections 2023-11-24T06:18:54.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3733835989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 0x7fa8d40b3d40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3733835989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d40b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3733835989 >> 172.21.15.89:0/3733835989 conn(0x7fa8d4071b90 msgr2=0x7fa8d4071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3733835989 shutdown_connections 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.279+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3733835989 wait complete. 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.281+0000 7fa8d98fc700 1 Processor -- start 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.281+0000 7fa8d98fc700 1 -- start start 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d98fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 0x7fa8d413a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d98fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8d40adf00 0x7fa8d40cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d98fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d4153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d98fc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40ceb80 con 0x7fa8d40adf00 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d98fc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40cee60 con 0x7fa8d406c710 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d98fc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40cf140 con 0x7fa8d41446a0 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d27fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8d40adf00 0x7fa8d40cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d27fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8d40adf00 0x7fa8d40cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35064/0 (socket says 172.21.15.89:35064) 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d27fc700 1 -- 172.21.15.89:0/3107221930 learned_addr learned my addr 172.21.15.89:0/3107221930 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d2ffd700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 0x7fa8d413a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d37fe700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d4153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d27fc700 1 -- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 msgr2=0x7fa8d4153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d27fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d4153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d27fc700 1 -- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 msgr2=0x7fa8d413a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d27fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 0x7fa8d413a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d27fc700 1 -- 172.21.15.89:0/3107221930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c800b7f0 con 0x7fa8d40adf00 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.282+0000 7fa8d2ffd700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 0x7fa8d413a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.283+0000 7fa8d27fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8d40adf00 0x7fa8d40cfc20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400c4f0 tx=0x7fa8c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.283+0000 7fa8bbfff700 1 -- 172.21.15.89:0/3107221930 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4005cb0 con 0x7fa8d40adf00 2023-11-24T06:18:54.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.283+0000 7fa8bbfff700 1 -- 172.21.15.89:0/3107221930 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c4005e10 con 0x7fa8d40adf00 2023-11-24T06:18:54.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.283+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc009a70 con 0x7fa8d40adf00 2023-11-24T06:18:54.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.283+0000 7fa8bbfff700 1 -- 172.21.15.89:0/3107221930 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40072d0 con 0x7fa8d40adf00 2023-11-24T06:18:54.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.283+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d40cf700 con 0x7fa8d40adf00 2023-11-24T06:18:54.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.284+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d40753f0 con 0x7fa8d40adf00 2023-11-24T06:18:54.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.284+0000 7fa8bbfff700 1 -- 172.21.15.89:0/3107221930 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa8c4007430 con 0x7fa8d40adf00 2023-11-24T06:18:54.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.285+0000 7fa8bbfff700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8c006aed0 0x7fa8c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.285+0000 7fa8d2ffd700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8c006aed0 0x7fa8c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.285+0000 7fa8bbfff700 1 -- 172.21.15.89:0/3107221930 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8c4088990 con 0x7fa8d40adf00 2023-11-24T06:18:54.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.285+0000 7fa8d2ffd700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8c006aed0 0x7fa8c006d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800be90 tx=0x7fa8c800b4d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:54.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.289+0000 7fa8bbfff700 1 -- 172.21.15.89:0/3107221930 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8c4053b00 con 0x7fa8d40adf00 2023-11-24T06:18:54.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.474+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8d40b6020 con 0x7fa8c006aed0 2023-11-24T06:18:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.481+0000 7fa8bbfff700 1 -- 172.21.15.89:0/3107221930 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264659 (secure 0 0 0) 0x7fa8c8015070 con 0x7fa8c006aed0 2023-11-24T06:18:54.486 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:54.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8c006aed0 msgr2=0x7fa8c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8c006aed0 0x7fa8c006d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800be90 tx=0x7fa8c800b4d0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8d40adf00 msgr2=0x7fa8d40cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8d40adf00 0x7fa8d40cfc20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400c4f0 tx=0x7fa8c400c8b0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 shutdown_connections 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8d41446a0 0x7fa8d4153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8c006aed0 0x7fa8c006d390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8d406c710 0x7fa8d413a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 --2- 172.21.15.89:0/3107221930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8d40adf00 0x7fa8d40cfc20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.484+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 >> 172.21.15.89:0/3107221930 conn(0x7fa8d4071b90 msgr2=0x7fa8d406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.485+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 shutdown_connections 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.485+0000 7fa8d98fc700 1 -- 172.21.15.89:0/3107221930 wait complete. 2023-11-24T06:18:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:18:54.498 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-11-24T06:18:53.718677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2848,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2916352,"data_stored":1752404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10],"upper_bound":16},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.730641"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024214+0000","last_change":"2023-11-24T06:18:50.033723+0000","last_active":"2023-11-24T06:18:51.024214+0000","last_peered":"2023-11-24T06:18:51.024214+0000","last_clean":"2023-11-24T06:18:51.024214+0000","last_became_active":"2023-11-24T06:18:50.033433+0000","last_became_peered":"2023-11-24T06:18:50.033433+0000","last_unstale":"2023-11-24T06:18:51.024214+0000","last_undegraded":"2023-11-24T06:18:51.024214+0000","last_fullsized":"2023-11-24T06:18:51.024214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:49.822079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021557+0000","last_change":"2023-11-24T06:18:50.024643+0000","last_active":"2023-11-24T06:18:51.021557+0000","last_peered":"2023-11-24T06:18:51.021557+0000","last_clean":"2023-11-24T06:18:51.021557+0000","last_became_active":"2023-11-24T06:18:50.024352+0000","last_became_peered":"2023-11-24T06:18:50.024352+0000","last_unstale":"2023-11-24T06:18:51.021557+0000","last_undegraded":"2023-11-24T06:18:51.021557+0000","last_fullsized":"2023-11-24T06:18:51.021557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:59.327185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032185+0000","last_change":"2023-11-24T06:18:50.025637+0000","last_active":"2023-11-24T06:18:50.032185+0000","last_peered":"2023-11-24T06:18:50.032185+0000","last_clean":"2023-11-24T06:18:50.032185+0000","last_became_active":"2023-11-24T06:18:50.025328+0000","last_became_peered":"2023-11-24T06:18:50.025328+0000","last_unstale":"2023-11-24T06:18:50.032185+0000","last_undegraded":"2023-11-24T06:18:50.032185+0000","last_fullsized":"2023-11-24T06:18:50.032185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.039592+0000","last_change":"2023-11-24T06:18:50.030667+0000","last_active":"2023-11-24T06:18:50.039592+0000","last_peered":"2023-11-24T06:18:50.039592+0000","last_clean":"2023-11-24T06:18:50.039592+0000","last_became_active":"2023-11-24T06:18:50.030133+0000","last_became_peered":"2023-11-24T06:18:50.030133+0000","last_unstale":"2023-11-24T06:18:50.039592+0000","last_undegraded":"2023-11-24T06:18:50.039592+0000","last_fullsized":"2023-11-24T06:18:50.039592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024116+0000","last_change":"2023-11-24T06:18:50.025413+0000","last_active":"2023-11-24T06:18:51.024116+0000","last_peered":"2023-11-24T06:18:51.024116+0000","last_clean":"2023-11-24T06:18:51.024116+0000","last_became_active":"2023-11-24T06:18:50.025118+0000","last_became_peered":"2023-11-24T06:18:50.025118+0000","last_unstale":"2023-11-24T06:18:51.024116+0000","last_undegraded":"2023-11-24T06:18:51.024116+0000","last_fullsized":"2023-11-24T06:18:51.024116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:21.299524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041485+0000","last_change":"2023-11-24T06:18:50.039892+0000","last_active":"2023-11-24T06:18:50.041485+0000","last_peered":"2023-11-24T06:18:50.041485+0000","last_clean":"2023-11-24T06:18:50.041485+0000","last_became_active":"2023-11-24T06:18:50.039525+0000","last_became_peered":"2023-11-24T06:18:50.039525+0000","last_unstale":"2023-11-24T06:18:50.041485+0000","last_undegraded":"2023-11-24T06:18:50.041485+0000","last_fullsized":"2023-11-24T06:18:50.041485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:41.341947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022843+0000","last_change":"2023-11-24T06:18:50.024213+0000","last_active":"2023-11-24T06:18:51.022843+0000","last_peered":"2023-11-24T06:18:51.022843+0000","last_clean":"2023-11-24T06:18:51.022843+0000","last_became_active":"2023-11-24T06:18:50.023914+0000","last_became_peered":"2023-11-24T06:18:50.023914+0000","last_unstale":"2023-11-24T06:18:51.022843+0000","last_undegraded":"2023-11-24T06:18:51.022843+0000","last_fullsized":"2023-11-24T06:18:51.022843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:48.527966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038582+0000","last_change":"2023-11-24T06:18:50.037907+0000","last_active":"2023-11-24T06:18:50.038582+0000","last_peered":"2023-11-24T06:18:50.038582+0000","last_clean":"2023-11-24T06:18:50.038582+0000","last_became_active":"2023-11-24T06:18:50.037606+0000","last_became_peered":"2023-11-24T06:18:50.037606+0000","last_unstale":"2023-11-24T06:18:50.038582+0000","last_undegraded":"2023-11-24T06:18:50.038582+0000","last_fullsized":"2023-11-24T06:18:50.038582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024189+0000","last_change":"2023-11-24T06:18:50.035382+0000","last_active":"2023-11-24T06:18:51.024189+0000","last_peered":"2023-11-24T06:18:51.024189+0000","last_clean":"2023-11-24T06:18:51.024189+0000","last_became_active":"2023-11-24T06:18:50.035148+0000","last_became_peered":"2023-11-24T06:18:50.035148+0000","last_unstale":"2023-11-24T06:18:51.024189+0000","last_undegraded":"2023-11-24T06:18:51.024189+0000","last_fullsized":"2023-11-24T06:18:51.024189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:39.141407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.039953+0000","last_change":"2023-11-24T06:18:50.037425+0000","last_active":"2023-11-24T06:18:50.039953+0000","last_peered":"2023-11-24T06:18:50.039953+0000","last_clean":"2023-11-24T06:18:50.039953+0000","last_became_active":"2023-11-24T06:18:50.037111+0000","last_became_peered":"2023-11-24T06:18:50.037111+0000","last_unstale":"2023-11-24T06:18:50.039953+0000","last_undegraded":"2023-11-24T06:18:50.039953+0000","last_fullsized":"2023-11-24T06:18:50.039953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:25.578418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040281+0000","last_change":"2023-11-24T06:18:50.036955+0000","last_active":"2023-11-24T06:18:50.040281+0000","last_peered":"2023-11-24T06:18:50.040281+0000","last_clean":"2023-11-24T06:18:50.040281+0000","last_became_active":"2023-11-24T06:18:50.036643+0000","last_became_peered":"2023-11-24T06:18:50.036643+0000","last_unstale":"2023-11-24T06:18:50.040281+0000","last_undegraded":"2023-11-24T06:18:50.040281+0000","last_fullsized":"2023-11-24T06:18:50.040281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:41.341947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023821+0000","last_change":"2023-11-24T06:18:50.034218+0000","last_active":"2023-11-24T06:18:51.023821+0000","last_peered":"2023-11-24T06:18:51.023821+0000","last_clean":"2023-11-24T06:18:51.023821+0000","last_became_active":"2023-11-24T06:18:50.033754+0000","last_became_peered":"2023-11-24T06:18:50.033754+0000","last_unstale":"2023-11-24T06:18:51.023821+0000","last_undegraded":"2023-11-24T06:18:51.023821+0000","last_fullsized":"2023-11-24T06:18:51.023821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036464+0000","last_change":"2023-11-24T06:18:50.033709+0000","last_active":"2023-11-24T06:18:50.036464+0000","last_peered":"2023-11-24T06:18:50.036464+0000","last_clean":"2023-11-24T06:18:50.036464+0000","last_became_active":"2023-11-24T06:18:50.033447+0000","last_became_peered":"2023-11-24T06:18:50.033447+0000","last_unstale":"2023-11-24T06:18:50.036464+0000","last_undegraded":"2023-11-24T06:18:50.036464+0000","last_fullsized":"2023-11-24T06:18:50.036464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:25.578418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.031860+0000","last_change":"2023-11-24T06:18:50.023134+0000","last_active":"2023-11-24T06:18:50.031860+0000","last_peered":"2023-11-24T06:18:50.031860+0000","last_clean":"2023-11-24T06:18:50.031860+0000","last_became_active":"2023-11-24T06:18:50.022837+0000","last_became_peered":"2023-11-24T06:18:50.022837+0000","last_unstale":"2023-11-24T06:18:50.031860+0000","last_undegraded":"2023-11-24T06:18:50.031860+0000","last_fullsized":"2023-11-24T06:18:50.031860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:27.849578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021102+0000","last_change":"2023-11-24T06:18:50.027089+0000","last_active":"2023-11-24T06:18:51.021102+0000","last_peered":"2023-11-24T06:18:51.021102+0000","last_clean":"2023-11-24T06:18:51.021102+0000","last_became_active":"2023-11-24T06:18:50.026810+0000","last_became_peered":"2023-11-24T06:18:50.026810+0000","last_unstale":"2023-11-24T06:18:51.021102+0000","last_undegraded":"2023-11-24T06:18:51.021102+0000","last_fullsized":"2023-11-24T06:18:51.021102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035433+0000","last_change":"2023-11-24T06:18:50.028542+0000","last_active":"2023-11-24T06:18:50.035433+0000","last_peered":"2023-11-24T06:18:50.035433+0000","last_clean":"2023-11-24T06:18:50.035433+0000","last_became_active":"2023-11-24T06:18:50.027971+0000","last_became_peered":"2023-11-24T06:18:50.027971+0000","last_unstale":"2023-11-24T06:18:50.035433+0000","last_undegraded":"2023-11-24T06:18:50.035433+0000","last_fullsized":"2023-11-24T06:18:50.035433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:27.849578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038210+0000","last_change":"2023-11-24T06:18:50.035313+0000","last_active":"2023-11-24T06:18:50.038210+0000","last_peered":"2023-11-24T06:18:50.038210+0000","last_clean":"2023-11-24T06:18:50.038210+0000","last_became_active":"2023-11-24T06:18:50.034744+0000","last_became_peered":"2023-11-24T06:18:50.034744+0000","last_unstale":"2023-11-24T06:18:50.038210+0000","last_undegraded":"2023-11-24T06:18:50.038210+0000","last_fullsized":"2023-11-24T06:18:50.038210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:34.485750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036447+0000","last_change":"2023-11-24T06:18:50.034072+0000","last_active":"2023-11-24T06:18:50.036447+0000","last_peered":"2023-11-24T06:18:50.036447+0000","last_clean":"2023-11-24T06:18:50.036447+0000","last_became_active":"2023-11-24T06:18:50.033798+0000","last_became_peered":"2023-11-24T06:18:50.033798+0000","last_unstale":"2023-11-24T06:18:50.036447+0000","last_undegraded":"2023-11-24T06:18:50.036447+0000","last_fullsized":"2023-11-24T06:18:50.036447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023796+0000","last_change":"2023-11-24T06:18:50.025458+0000","last_active":"2023-11-24T06:18:51.023796+0000","last_peered":"2023-11-24T06:18:51.023796+0000","last_clean":"2023-11-24T06:18:51.023796+0000","last_became_active":"2023-11-24T06:18:50.025139+0000","last_became_peered":"2023-11-24T06:18:50.025139+0000","last_unstale":"2023-11-24T06:18:51.023796+0000","last_undegraded":"2023-11-24T06:18:51.023796+0000","last_fullsized":"2023-11-24T06:18:51.023796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:02:29.652618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.037010+0000","last_change":"2023-11-24T06:18:49.009105+0000","last_active":"2023-11-24T06:18:50.037010+0000","last_peered":"2023-11-24T06:18:50.037010+0000","last_clean":"2023-11-24T06:18:50.037010+0000","last_became_active":"2023-11-24T06:18:49.008802+0000","last_became_peered":"2023-11-24T06:18:49.008802+0000","last_unstale":"2023-11-24T06:18:50.037010+0000","last_undegraded":"2023-11-24T06:18:50.037010+0000","last_fullsized":"2023-11-24T06:18:50.037010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036544+0000","last_change":"2023-11-24T06:18:50.033700+0000","last_active":"2023-11-24T06:18:50.036544+0000","last_peered":"2023-11-24T06:18:50.036544+0000","last_clean":"2023-11-24T06:18:50.036544+0000","last_became_active":"2023-11-24T06:18:50.033372+0000","last_became_peered":"2023-11-24T06:18:50.033372+0000","last_unstale":"2023-11-24T06:18:50.036544+0000","last_undegraded":"2023-11-24T06:18:50.036544+0000","last_fullsized":"2023-11-24T06:18:50.036544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036391+0000","last_change":"2023-11-24T06:18:49.015649+0000","last_active":"2023-11-24T06:18:50.036391+0000","last_peered":"2023-11-24T06:18:50.036391+0000","last_clean":"2023-11-24T06:18:50.036391+0000","last_became_active":"2023-11-24T06:18:49.015329+0000","last_became_peered":"2023-11-24T06:18:49.015329+0000","last_unstale":"2023-11-24T06:18:50.036391+0000","last_undegraded":"2023-11-24T06:18:50.036391+0000","last_fullsized":"2023-11-24T06:18:50.036391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038969+0000","last_change":"2023-11-24T06:18:50.037859+0000","last_active":"2023-11-24T06:18:50.038969+0000","last_peered":"2023-11-24T06:18:50.038969+0000","last_clean":"2023-11-24T06:18:50.038969+0000","last_became_active":"2023-11-24T06:18:50.037565+0000","last_became_peered":"2023-11-24T06:18:50.037565+0000","last_unstale":"2023-11-24T06:18:50.038969+0000","last_undegraded":"2023-11-24T06:18:50.038969+0000","last_fullsized":"2023-11-24T06:18:50.038969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024207+0000","last_change":"2023-11-24T06:18:49.008317+0000","last_active":"2023-11-24T06:18:51.024207+0000","last_peered":"2023-11-24T06:18:51.024207+0000","last_clean":"2023-11-24T06:18:51.024207+0000","last_became_active":"2023-11-24T06:18:49.008039+0000","last_became_peered":"2023-11-24T06:18:49.008039+0000","last_unstale":"2023-11-24T06:18:51.024207+0000","last_undegraded":"2023-11-24T06:18:51.024207+0000","last_fullsized":"2023-11-24T06:18:51.024207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:58.320250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.039648+0000","last_change":"2023-11-24T06:18:50.034882+0000","last_active":"2023-11-24T06:18:50.039648+0000","last_peered":"2023-11-24T06:18:50.039648+0000","last_clean":"2023-11-24T06:18:50.039648+0000","last_became_active":"2023-11-24T06:18:50.031623+0000","last_became_peered":"2023-11-24T06:18:50.031623+0000","last_unstale":"2023-11-24T06:18:50.039648+0000","last_undegraded":"2023-11-24T06:18:50.039648+0000","last_fullsized":"2023-11-24T06:18:50.039648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:54.282518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023720+0000","last_change":"2023-11-24T06:18:49.007346+0000","last_active":"2023-11-24T06:18:51.023720+0000","last_peered":"2023-11-24T06:18:51.023720+0000","last_clean":"2023-11-24T06:18:51.023720+0000","last_became_active":"2023-11-24T06:18:49.006300+0000","last_became_peered":"2023-11-24T06:18:49.006300+0000","last_unstale":"2023-11-24T06:18:51.023720+0000","last_undegraded":"2023-11-24T06:18:51.023720+0000","last_fullsized":"2023-11-24T06:18:51.023720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.037998+0000","last_change":"2023-11-24T06:18:50.029883+0000","last_active":"2023-11-24T06:18:50.037998+0000","last_peered":"2023-11-24T06:18:50.037998+0000","last_clean":"2023-11-24T06:18:50.037998+0000","last_became_active":"2023-11-24T06:18:50.029618+0000","last_became_peered":"2023-11-24T06:18:50.029618+0000","last_unstale":"2023-11-24T06:18:50.037998+0000","last_undegraded":"2023-11-24T06:18:50.037998+0000","last_fullsized":"2023-11-24T06:18:50.037998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036344+0000","last_change":"2023-11-24T06:18:49.015809+0000","last_active":"2023-11-24T06:18:50.036344+0000","last_peered":"2023-11-24T06:18:50.036344+0000","last_clean":"2023-11-24T06:18:50.036344+0000","last_became_active":"2023-11-24T06:18:49.015073+0000","last_became_peered":"2023-11-24T06:18:49.015073+0000","last_unstale":"2023-11-24T06:18:50.036344+0000","last_undegraded":"2023-11-24T06:18:50.036344+0000","last_fullsized":"2023-11-24T06:18:50.036344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:40.335012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023148+0000","last_change":"2023-11-24T06:18:50.015301+0000","last_active":"2023-11-24T06:18:51.023148+0000","last_peered":"2023-11-24T06:18:51.023148+0000","last_clean":"2023-11-24T06:18:51.023148+0000","last_became_active":"2023-11-24T06:18:50.015007+0000","last_became_peered":"2023-11-24T06:18:50.015007+0000","last_unstale":"2023-11-24T06:18:51.023148+0000","last_undegraded":"2023-11-24T06:18:51.023148+0000","last_fullsized":"2023-11-24T06:18:51.023148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.033470+0000","last_change":"2023-11-24T06:18:49.008047+0000","last_active":"2023-11-24T06:18:50.033470+0000","last_peered":"2023-11-24T06:18:50.033470+0000","last_clean":"2023-11-24T06:18:50.033470+0000","last_became_active":"2023-11-24T06:18:49.007764+0000","last_became_peered":"2023-11-24T06:18:49.007764+0000","last_unstale":"2023-11-24T06:18:50.033470+0000","last_undegraded":"2023-11-24T06:18:50.033470+0000","last_fullsized":"2023-11-24T06:18:50.033470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:02.607697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032842+0000","last_change":"2023-11-24T06:18:50.027945+0000","last_active":"2023-11-24T06:18:50.032842+0000","last_peered":"2023-11-24T06:18:50.032842+0000","last_clean":"2023-11-24T06:18:50.032842+0000","last_became_active":"2023-11-24T06:18:50.027609+0000","last_became_peered":"2023-11-24T06:18:50.027609+0000","last_unstale":"2023-11-24T06:18:50.032842+0000","last_undegraded":"2023-11-24T06:18:50.032842+0000","last_fullsized":"2023-11-24T06:18:50.032842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:52.017798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.019687+0000","last_change":"2023-11-24T06:18:49.014611+0000","last_active":"2023-11-24T06:18:50.019687+0000","last_peered":"2023-11-24T06:18:50.019687+0000","last_clean":"2023-11-24T06:18:50.019687+0000","last_became_active":"2023-11-24T06:18:49.014277+0000","last_became_peered":"2023-11-24T06:18:49.014277+0000","last_unstale":"2023-11-24T06:18:50.019687+0000","last_undegraded":"2023-11-24T06:18:50.019687+0000","last_fullsized":"2023-11-24T06:18:50.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:33.478815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035905+0000","last_change":"2023-11-24T06:18:50.027283+0000","last_active":"2023-11-24T06:18:50.035905+0000","last_peered":"2023-11-24T06:18:50.035905+0000","last_clean":"2023-11-24T06:18:50.035905+0000","last_became_active":"2023-11-24T06:18:50.027028+0000","last_became_peered":"2023-11-24T06:18:50.027028+0000","last_unstale":"2023-11-24T06:18:50.035905+0000","last_undegraded":"2023-11-24T06:18:50.035905+0000","last_fullsized":"2023-11-24T06:18:50.035905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:24.709998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.019779+0000","last_change":"2023-11-24T06:18:49.007327+0000","last_active":"2023-11-24T06:18:50.019779+0000","last_peered":"2023-11-24T06:18:50.019779+0000","last_clean":"2023-11-24T06:18:50.019779+0000","last_became_active":"2023-11-24T06:18:49.006684+0000","last_became_peered":"2023-11-24T06:18:49.006684+0000","last_unstale":"2023-11-24T06:18:50.019779+0000","last_undegraded":"2023-11-24T06:18:50.019779+0000","last_fullsized":"2023-11-24T06:18:50.019779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:54.335883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021189+0000","last_change":"2023-11-24T06:18:50.026294+0000","last_active":"2023-11-24T06:18:51.021189+0000","last_peered":"2023-11-24T06:18:51.021189+0000","last_clean":"2023-11-24T06:18:51.021189+0000","last_became_active":"2023-11-24T06:18:50.026020+0000","last_became_peered":"2023-11-24T06:18:50.026020+0000","last_unstale":"2023-11-24T06:18:51.021189+0000","last_undegraded":"2023-11-24T06:18:51.021189+0000","last_fullsized":"2023-11-24T06:18:51.021189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.017768+0000","last_change":"2023-11-24T06:18:49.011149+0000","last_active":"2023-11-24T06:18:50.017768+0000","last_peered":"2023-11-24T06:18:50.017768+0000","last_clean":"2023-11-24T06:18:50.017768+0000","last_became_active":"2023-11-24T06:18:49.010573+0000","last_became_peered":"2023-11-24T06:18:49.010573+0000","last_unstale":"2023-11-24T06:18:50.017768+0000","last_undegraded":"2023-11-24T06:18:50.017768+0000","last_fullsized":"2023-11-24T06:18:50.017768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040771+0000","last_change":"2023-11-24T06:18:50.035496+0000","last_active":"2023-11-24T06:18:50.040771+0000","last_peered":"2023-11-24T06:18:50.040771+0000","last_clean":"2023-11-24T06:18:50.040771+0000","last_became_active":"2023-11-24T06:18:50.035172+0000","last_became_peered":"2023-11-24T06:18:50.035172+0000","last_unstale":"2023-11-24T06:18:50.040771+0000","last_undegraded":"2023-11-24T06:18:50.040771+0000","last_fullsized":"2023-11-24T06:18:50.040771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:56:59.481984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.014777+0000","last_change":"2023-11-24T06:18:49.012031+0000","last_active":"2023-11-24T06:18:50.014777+0000","last_peered":"2023-11-24T06:18:50.014777+0000","last_clean":"2023-11-24T06:18:50.014777+0000","last_became_active":"2023-11-24T06:18:49.011741+0000","last_became_peered":"2023-11-24T06:18:49.011741+0000","last_unstale":"2023-11-24T06:18:50.014777+0000","last_undegraded":"2023-11-24T06:18:50.014777+0000","last_fullsized":"2023-11-24T06:18:50.014777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038700+0000","last_change":"2023-11-24T06:18:50.035543+0000","last_active":"2023-11-24T06:18:50.038700+0000","last_peered":"2023-11-24T06:18:50.038700+0000","last_clean":"2023-11-24T06:18:50.038700+0000","last_became_active":"2023-11-24T06:18:50.035269+0000","last_became_peered":"2023-11-24T06:18:50.035269+0000","last_unstale":"2023-11-24T06:18:50.038700+0000","last_undegraded":"2023-11-24T06:18:50.038700+0000","last_fullsized":"2023-11-24T06:18:50.038700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021480+0000","last_change":"2023-11-24T06:18:49.006503+0000","last_active":"2023-11-24T06:18:51.021480+0000","last_peered":"2023-11-24T06:18:51.021480+0000","last_clean":"2023-11-24T06:18:51.021480+0000","last_became_active":"2023-11-24T06:18:49.006176+0000","last_became_peered":"2023-11-24T06:18:49.006176+0000","last_unstale":"2023-11-24T06:18:51.021480+0000","last_undegraded":"2023-11-24T06:18:51.021480+0000","last_fullsized":"2023-11-24T06:18:51.021480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:33.478815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023461+0000","last_change":"2023-11-24T06:18:50.023420+0000","last_active":"2023-11-24T06:18:51.023461+0000","last_peered":"2023-11-24T06:18:51.023461+0000","last_clean":"2023-11-24T06:18:51.023461+0000","last_became_active":"2023-11-24T06:18:50.022722+0000","last_became_peered":"2023-11-24T06:18:50.022722+0000","last_unstale":"2023-11-24T06:18:51.023461+0000","last_undegraded":"2023-11-24T06:18:51.023461+0000","last_fullsized":"2023-11-24T06:18:51.023461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:24.709998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040671+0000","last_change":"2023-11-24T06:18:49.016525+0000","last_active":"2023-11-24T06:18:50.040671+0000","last_peered":"2023-11-24T06:18:50.040671+0000","last_clean":"2023-11-24T06:18:50.040671+0000","last_became_active":"2023-11-24T06:18:49.016259+0000","last_became_peered":"2023-11-24T06:18:49.016259+0000","last_unstale":"2023-11-24T06:18:50.040671+0000","last_undegraded":"2023-11-24T06:18:50.040671+0000","last_fullsized":"2023-11-24T06:18:50.040671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:13.224297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023938+0000","last_change":"2023-11-24T06:18:50.024377+0000","last_active":"2023-11-24T06:18:51.023938+0000","last_peered":"2023-11-24T06:18:51.023938+0000","last_clean":"2023-11-24T06:18:51.023938+0000","last_became_active":"2023-11-24T06:18:50.024123+0000","last_became_peered":"2023-11-24T06:18:50.024123+0000","last_unstale":"2023-11-24T06:18:51.023938+0000","last_undegraded":"2023-11-24T06:18:51.023938+0000","last_fullsized":"2023-11-24T06:18:51.023938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:28:39.352489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041197+0000","last_change":"2023-11-24T06:18:49.009176+0000","last_active":"2023-11-24T06:18:50.041197+0000","last_peered":"2023-11-24T06:18:50.041197+0000","last_clean":"2023-11-24T06:18:50.041197+0000","last_became_active":"2023-11-24T06:18:49.008875+0000","last_became_peered":"2023-11-24T06:18:49.008875+0000","last_unstale":"2023-11-24T06:18:50.041197+0000","last_undegraded":"2023-11-24T06:18:50.041197+0000","last_fullsized":"2023-11-24T06:18:50.041197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.034300+0000","last_change":"2023-11-24T06:18:50.032393+0000","last_active":"2023-11-24T06:18:50.034300+0000","last_peered":"2023-11-24T06:18:50.034300+0000","last_clean":"2023-11-24T06:18:50.034300+0000","last_became_active":"2023-11-24T06:18:50.032145+0000","last_became_peered":"2023-11-24T06:18:50.032145+0000","last_unstale":"2023-11-24T06:18:50.034300+0000","last_undegraded":"2023-11-24T06:18:50.034300+0000","last_fullsized":"2023-11-24T06:18:50.034300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.020491+0000","last_change":"2023-11-24T06:18:49.015579+0000","last_active":"2023-11-24T06:18:50.020491+0000","last_peered":"2023-11-24T06:18:50.020491+0000","last_clean":"2023-11-24T06:18:50.020491+0000","last_became_active":"2023-11-24T06:18:49.015329+0000","last_became_peered":"2023-11-24T06:18:49.015329+0000","last_unstale":"2023-11-24T06:18:50.020491+0000","last_undegraded":"2023-11-24T06:18:50.020491+0000","last_fullsized":"2023-11-24T06:18:50.020491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:37.250753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024128+0000","last_change":"2023-11-24T06:18:50.021964+0000","last_active":"2023-11-24T06:18:51.024128+0000","last_peered":"2023-11-24T06:18:51.024128+0000","last_clean":"2023-11-24T06:18:51.024128+0000","last_became_active":"2023-11-24T06:18:50.021678+0000","last_became_peered":"2023-11-24T06:18:50.021678+0000","last_unstale":"2023-11-24T06:18:51.024128+0000","last_undegraded":"2023-11-24T06:18:51.024128+0000","last_fullsized":"2023-11-24T06:18:51.024128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:32:20.935789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.017027+0000","last_change":"2023-11-24T06:18:49.007859+0000","last_active":"2023-11-24T06:18:50.017027+0000","last_peered":"2023-11-24T06:18:50.017027+0000","last_clean":"2023-11-24T06:18:50.017027+0000","last_became_active":"2023-11-24T06:18:49.007569+0000","last_became_peered":"2023-11-24T06:18:49.007569+0000","last_unstale":"2023-11-24T06:18:50.017027+0000","last_undegraded":"2023-11-24T06:18:50.017027+0000","last_fullsized":"2023-11-24T06:18:50.017027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038984+0000","last_change":"2023-11-24T06:18:50.033953+0000","last_active":"2023-11-24T06:18:50.038984+0000","last_peered":"2023-11-24T06:18:50.038984+0000","last_clean":"2023-11-24T06:18:50.038984+0000","last_became_active":"2023-11-24T06:18:50.033664+0000","last_became_peered":"2023-11-24T06:18:50.033664+0000","last_unstale":"2023-11-24T06:18:50.038984+0000","last_undegraded":"2023-11-24T06:18:50.038984+0000","last_fullsized":"2023-11-24T06:18:50.038984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:54.282518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.037164+0000","last_change":"2023-11-24T06:18:49.009433+0000","last_active":"2023-11-24T06:18:50.037164+0000","last_peered":"2023-11-24T06:18:50.037164+0000","last_clean":"2023-11-24T06:18:50.037164+0000","last_became_active":"2023-11-24T06:18:49.009133+0000","last_became_peered":"2023-11-24T06:18:49.009133+0000","last_unstale":"2023-11-24T06:18:50.037164+0000","last_undegraded":"2023-11-24T06:18:50.037164+0000","last_fullsized":"2023-11-24T06:18:50.037164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023055+0000","last_change":"2023-11-24T06:18:50.023048+0000","last_active":"2023-11-24T06:18:51.023055+0000","last_peered":"2023-11-24T06:18:51.023055+0000","last_clean":"2023-11-24T06:18:51.023055+0000","last_became_active":"2023-11-24T06:18:50.022736+0000","last_became_peered":"2023-11-24T06:18:50.022736+0000","last_unstale":"2023-11-24T06:18:51.023055+0000","last_undegraded":"2023-11-24T06:18:51.023055+0000","last_fullsized":"2023-11-24T06:18:51.023055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:14.887594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023699+0000","last_change":"2023-11-24T06:18:49.012343+0000","last_active":"2023-11-24T06:18:51.023699+0000","last_peered":"2023-11-24T06:18:51.023699+0000","last_clean":"2023-11-24T06:18:51.023699+0000","last_became_active":"2023-11-24T06:18:49.012078+0000","last_became_peered":"2023-11-24T06:18:49.012078+0000","last_unstale":"2023-11-24T06:18:51.023699+0000","last_undegraded":"2023-11-24T06:18:51.023699+0000","last_fullsized":"2023-11-24T06:18:51.023699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024206+0000","last_change":"2023-11-24T06:18:50.022294+0000","last_active":"2023-11-24T06:18:51.024206+0000","last_peered":"2023-11-24T06:18:51.024206+0000","last_clean":"2023-11-24T06:18:51.024206+0000","last_became_active":"2023-11-24T06:18:50.021806+0000","last_became_peered":"2023-11-24T06:18:50.021806+0000","last_unstale":"2023-11-24T06:18:51.024206+0000","last_undegraded":"2023-11-24T06:18:51.024206+0000","last_fullsized":"2023-11-24T06:18:51.024206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:03.577619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021419+0000","last_change":"2023-11-24T06:18:49.010001+0000","last_active":"2023-11-24T06:18:51.021419+0000","last_peered":"2023-11-24T06:18:51.021419+0000","last_clean":"2023-11-24T06:18:51.021419+0000","last_became_active":"2023-11-24T06:18:49.009729+0000","last_became_peered":"2023-11-24T06:18:49.009729+0000","last_unstale":"2023-11-24T06:18:51.021419+0000","last_undegraded":"2023-11-24T06:18:51.021419+0000","last_fullsized":"2023-11-24T06:18:51.021419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:24.571482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040814+0000","last_change":"2023-11-24T06:18:50.035809+0000","last_active":"2023-11-24T06:18:50.040814+0000","last_peered":"2023-11-24T06:18:50.040814+0000","last_clean":"2023-11-24T06:18:50.040814+0000","last_became_active":"2023-11-24T06:18:50.033211+0000","last_became_peered":"2023-11-24T06:18:50.033211+0000","last_unstale":"2023-11-24T06:18:50.040814+0000","last_undegraded":"2023-11-24T06:18:50.040814+0000","last_fullsized":"2023-11-24T06:18:50.040814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.023048+0000","last_change":"2023-11-24T06:18:49.004923+0000","last_active":"2023-11-24T06:18:50.023048+0000","last_peered":"2023-11-24T06:18:50.023048+0000","last_clean":"2023-11-24T06:18:50.023048+0000","last_became_active":"2023-11-24T06:18:49.004601+0000","last_became_peered":"2023-11-24T06:18:49.004601+0000","last_unstale":"2023-11-24T06:18:50.023048+0000","last_undegraded":"2023-11-24T06:18:50.023048+0000","last_fullsized":"2023-11-24T06:18:50.023048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:42.514481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023365+0000","last_change":"2023-11-24T06:18:50.026574+0000","last_active":"2023-11-24T06:18:51.023365+0000","last_peered":"2023-11-24T06:18:51.023365+0000","last_clean":"2023-11-24T06:18:51.023365+0000","last_became_active":"2023-11-24T06:18:50.026028+0000","last_became_peered":"2023-11-24T06:18:50.026028+0000","last_unstale":"2023-11-24T06:18:51.023365+0000","last_undegraded":"2023-11-24T06:18:51.023365+0000","last_fullsized":"2023-11-24T06:18:51.023365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021109+0000","last_change":"2023-11-24T06:18:50.027544+0000","last_active":"2023-11-24T06:18:51.021109+0000","last_peered":"2023-11-24T06:18:51.021109+0000","last_clean":"2023-11-24T06:18:51.021109+0000","last_became_active":"2023-11-24T06:18:50.027272+0000","last_became_peered":"2023-11-24T06:18:50.027272+0000","last_unstale":"2023-11-24T06:18:51.021109+0000","last_undegraded":"2023-11-24T06:18:51.021109+0000","last_fullsized":"2023-11-24T06:18:51.021109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.019067+0000","last_change":"2023-11-24T06:18:49.015756+0000","last_active":"2023-11-24T06:18:50.019067+0000","last_peered":"2023-11-24T06:18:50.019067+0000","last_clean":"2023-11-24T06:18:50.019067+0000","last_became_active":"2023-11-24T06:18:49.015446+0000","last_became_peered":"2023-11-24T06:18:49.015446+0000","last_unstale":"2023-11-24T06:18:50.019067+0000","last_undegraded":"2023-11-24T06:18:50.019067+0000","last_fullsized":"2023-11-24T06:18:50.019067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:24.571482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.040735+0000","last_change":"2023-11-24T06:18:50.040018+0000","last_active":"2023-11-24T06:18:50.040735+0000","last_peered":"2023-11-24T06:18:50.040735+0000","last_clean":"2023-11-24T06:18:50.040735+0000","last_became_active":"2023-11-24T06:18:50.039786+0000","last_became_peered":"2023-11-24T06:18:50.039786+0000","last_unstale":"2023-11-24T06:18:50.040735+0000","last_undegraded":"2023-11-24T06:18:50.040735+0000","last_fullsized":"2023-11-24T06:18:50.040735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:27.849578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.014883+0000","last_change":"2023-11-24T06:18:49.009457+0000","last_active":"2023-11-24T06:18:50.014883+0000","last_peered":"2023-11-24T06:18:50.014883+0000","last_clean":"2023-11-24T06:18:50.014883+0000","last_became_active":"2023-11-24T06:18:49.008932+0000","last_became_peered":"2023-11-24T06:18:49.008932+0000","last_unstale":"2023-11-24T06:18:50.014883+0000","last_undegraded":"2023-11-24T06:18:50.014883+0000","last_fullsized":"2023-11-24T06:18:50.014883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.009997+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:18:50.009997+0000","last_peered":"2023-11-24T06:18:50.009997+0000","last_clean":"2023-11-24T06:18:50.009997+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:18:50.009997+0000","last_undegraded":"2023-11-24T06:18:50.009997+0000","last_fullsized":"2023-11-24T06:18:50.009997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032512+0000","last_change":"2023-11-24T06:18:50.027632+0000","last_active":"2023-11-24T06:18:50.032512+0000","last_peered":"2023-11-24T06:18:50.032512+0000","last_clean":"2023-11-24T06:18:50.032512+0000","last_became_active":"2023-11-24T06:18:50.027344+0000","last_became_peered":"2023-11-24T06:18:50.027344+0000","last_unstale":"2023-11-24T06:18:50.032512+0000","last_undegraded":"2023-11-24T06:18:50.032512+0000","last_fullsized":"2023-11-24T06:18:50.032512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:25.578418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022810+0000","last_change":"2023-11-24T06:18:49.006996+0000","last_active":"2023-11-24T06:18:51.022810+0000","last_peered":"2023-11-24T06:18:51.022810+0000","last_clean":"2023-11-24T06:18:51.022810+0000","last_became_active":"2023-11-24T06:18:49.006652+0000","last_became_peered":"2023-11-24T06:18:49.006652+0000","last_unstale":"2023-11-24T06:18:51.022810+0000","last_undegraded":"2023-11-24T06:18:51.022810+0000","last_fullsized":"2023-11-24T06:18:51.022810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:01.167887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024055+0000","last_change":"2023-11-24T06:18:49.012066+0000","last_active":"2023-11-24T06:18:51.024055+0000","last_peered":"2023-11-24T06:18:51.024055+0000","last_clean":"2023-11-24T06:18:51.024055+0000","last_became_active":"2023-11-24T06:18:49.011795+0000","last_became_peered":"2023-11-24T06:18:49.011795+0000","last_unstale":"2023-11-24T06:18:51.024055+0000","last_undegraded":"2023-11-24T06:18:51.024055+0000","last_fullsized":"2023-11-24T06:18:51.024055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:22.850843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021314+0000","last_change":"2023-11-24T06:18:50.021739+0000","last_active":"2023-11-24T06:18:51.021314+0000","last_peered":"2023-11-24T06:18:51.021314+0000","last_clean":"2023-11-24T06:18:51.021314+0000","last_became_active":"2023-11-24T06:18:50.021452+0000","last_became_peered":"2023-11-24T06:18:50.021452+0000","last_unstale":"2023-11-24T06:18:51.021314+0000","last_undegraded":"2023-11-24T06:18:51.021314+0000","last_fullsized":"2023-11-24T06:18:51.021314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021363+0000","last_change":"2023-11-24T06:18:49.012066+0000","last_active":"2023-11-24T06:18:51.021363+0000","last_peered":"2023-11-24T06:18:51.021363+0000","last_clean":"2023-11-24T06:18:51.021363+0000","last_became_active":"2023-11-24T06:18:49.011748+0000","last_became_peered":"2023-11-24T06:18:49.011748+0000","last_unstale":"2023-11-24T06:18:51.021363+0000","last_undegraded":"2023-11-24T06:18:51.021363+0000","last_fullsized":"2023-11-24T06:18:51.021363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036495+0000","last_change":"2023-11-24T06:18:50.033206+0000","last_active":"2023-11-24T06:18:50.036495+0000","last_peered":"2023-11-24T06:18:50.036495+0000","last_clean":"2023-11-24T06:18:50.036495+0000","last_became_active":"2023-11-24T06:18:50.032878+0000","last_became_peered":"2023-11-24T06:18:50.032878+0000","last_unstale":"2023-11-24T06:18:50.036495+0000","last_undegraded":"2023-11-24T06:18:50.036495+0000","last_fullsized":"2023-11-24T06:18:50.036495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:03.614632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036352+0000","last_change":"2023-11-24T06:18:49.011356+0000","last_active":"2023-11-24T06:18:50.036352+0000","last_peered":"2023-11-24T06:18:50.036352+0000","last_clean":"2023-11-24T06:18:50.036352+0000","last_became_active":"2023-11-24T06:18:49.010903+0000","last_became_peered":"2023-11-24T06:18:49.010903+0000","last_unstale":"2023-11-24T06:18:50.036352+0000","last_undegraded":"2023-11-24T06:18:50.036352+0000","last_fullsized":"2023-11-24T06:18:50.036352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032088+0000","last_change":"2023-11-24T06:18:50.024110+0000","last_active":"2023-11-24T06:18:50.032088+0000","last_peered":"2023-11-24T06:18:50.032088+0000","last_clean":"2023-11-24T06:18:50.032088+0000","last_became_active":"2023-11-24T06:18:50.023801+0000","last_became_peered":"2023-11-24T06:18:50.023801+0000","last_unstale":"2023-11-24T06:18:50.032088+0000","last_undegraded":"2023-11-24T06:18:50.032088+0000","last_fullsized":"2023-11-24T06:18:50.032088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:03.614632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022707+0000","last_change":"2023-11-24T06:18:49.010428+0000","last_active":"2023-11-24T06:18:51.022707+0000","last_peered":"2023-11-24T06:18:51.022707+0000","last_clean":"2023-11-24T06:18:51.022707+0000","last_became_active":"2023-11-24T06:18:49.010124+0000","last_became_peered":"2023-11-24T06:18:49.010124+0000","last_unstale":"2023-11-24T06:18:51.022707+0000","last_undegraded":"2023-11-24T06:18:51.022707+0000","last_fullsized":"2023-11-24T06:18:51.022707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023378+0000","last_change":"2023-11-24T06:18:50.022092+0000","last_active":"2023-11-24T06:18:51.023378+0000","last_peered":"2023-11-24T06:18:51.023378+0000","last_clean":"2023-11-24T06:18:51.023378+0000","last_became_active":"2023-11-24T06:18:50.021808+0000","last_became_peered":"2023-11-24T06:18:50.021808+0000","last_unstale":"2023-11-24T06:18:51.023378+0000","last_undegraded":"2023-11-24T06:18:51.023378+0000","last_fullsized":"2023-11-24T06:18:51.023378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:21.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021058+0000","last_change":"2023-11-24T06:18:49.007474+0000","last_active":"2023-11-24T06:18:51.021058+0000","last_peered":"2023-11-24T06:18:51.021058+0000","last_clean":"2023-11-24T06:18:51.021058+0000","last_became_active":"2023-11-24T06:18:49.007228+0000","last_became_peered":"2023-11-24T06:18:49.007228+0000","last_unstale":"2023-11-24T06:18:51.021058+0000","last_undegraded":"2023-11-24T06:18:51.021058+0000","last_fullsized":"2023-11-24T06:18:51.021058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035548+0000","last_change":"2023-11-24T06:18:50.025141+0000","last_active":"2023-11-24T06:18:50.035548+0000","last_peered":"2023-11-24T06:18:50.035548+0000","last_clean":"2023-11-24T06:18:50.035548+0000","last_became_active":"2023-11-24T06:18:50.024859+0000","last_became_peered":"2023-11-24T06:18:50.024859+0000","last_unstale":"2023-11-24T06:18:50.035548+0000","last_undegraded":"2023-11-24T06:18:50.035548+0000","last_fullsized":"2023-11-24T06:18:50.035548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021233+0000","last_change":"2023-11-24T06:18:49.015615+0000","last_active":"2023-11-24T06:18:51.021233+0000","last_peered":"2023-11-24T06:18:51.021233+0000","last_clean":"2023-11-24T06:18:51.021233+0000","last_became_active":"2023-11-24T06:18:49.015320+0000","last_became_peered":"2023-11-24T06:18:49.015320+0000","last_unstale":"2023-11-24T06:18:51.021233+0000","last_undegraded":"2023-11-24T06:18:51.021233+0000","last_fullsized":"2023-11-24T06:18:51.021233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035617+0000","last_change":"2023-11-24T06:18:50.025429+0000","last_active":"2023-11-24T06:18:50.035617+0000","last_peered":"2023-11-24T06:18:50.035617+0000","last_clean":"2023-11-24T06:18:50.035617+0000","last_became_active":"2023-11-24T06:18:50.024972+0000","last_became_peered":"2023-11-24T06:18:50.024972+0000","last_unstale":"2023-11-24T06:18:50.035617+0000","last_undegraded":"2023-11-24T06:18:50.035617+0000","last_fullsized":"2023-11-24T06:18:50.035617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:47:17.509901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024261+0000","last_change":"2023-11-24T06:18:49.007100+0000","last_active":"2023-11-24T06:18:51.024261+0000","last_peered":"2023-11-24T06:18:51.024261+0000","last_clean":"2023-11-24T06:18:51.024261+0000","last_became_active":"2023-11-24T06:18:49.006577+0000","last_became_peered":"2023-11-24T06:18:49.006577+0000","last_unstale":"2023-11-24T06:18:51.024261+0000","last_undegraded":"2023-11-24T06:18:51.024261+0000","last_fullsized":"2023-11-24T06:18:51.024261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.033566+0000","last_change":"2023-11-24T06:18:50.032437+0000","last_active":"2023-11-24T06:18:50.033566+0000","last_peered":"2023-11-24T06:18:50.033566+0000","last_clean":"2023-11-24T06:18:50.033566+0000","last_became_active":"2023-11-24T06:18:50.032082+0000","last_became_peered":"2023-11-24T06:18:50.032082+0000","last_unstale":"2023-11-24T06:18:50.033566+0000","last_undegraded":"2023-11-24T06:18:50.033566+0000","last_fullsized":"2023-11-24T06:18:50.033566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.019270+0000","last_change":"2023-11-24T06:18:49.011290+0000","last_active":"2023-11-24T06:18:50.019270+0000","last_peered":"2023-11-24T06:18:50.019270+0000","last_clean":"2023-11-24T06:18:50.019270+0000","last_became_active":"2023-11-24T06:18:49.010889+0000","last_became_peered":"2023-11-24T06:18:49.010889+0000","last_unstale":"2023-11-24T06:18:50.019270+0000","last_undegraded":"2023-11-24T06:18:50.019270+0000","last_fullsized":"2023-11-24T06:18:50.019270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024098+0000","last_change":"2023-11-24T06:18:50.036167+0000","last_active":"2023-11-24T06:18:51.024098+0000","last_peered":"2023-11-24T06:18:51.024098+0000","last_clean":"2023-11-24T06:18:51.024098+0000","last_became_active":"2023-11-24T06:18:50.035904+0000","last_became_peered":"2023-11-24T06:18:50.035904+0000","last_unstale":"2023-11-24T06:18:51.024098+0000","last_undegraded":"2023-11-24T06:18:51.024098+0000","last_fullsized":"2023-11-24T06:18:51.024098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:33:00.547188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021448+0000","last_change":"2023-11-24T06:18:49.006925+0000","last_active":"2023-11-24T06:18:51.021448+0000","last_peered":"2023-11-24T06:18:51.021448+0000","last_clean":"2023-11-24T06:18:51.021448+0000","last_became_active":"2023-11-24T06:18:49.006482+0000","last_became_peered":"2023-11-24T06:18:49.006482+0000","last_unstale":"2023-11-24T06:18:51.021448+0000","last_undegraded":"2023-11-24T06:18:51.021448+0000","last_fullsized":"2023-11-24T06:18:51.021448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:40.335012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041428+0000","last_change":"2023-11-24T06:18:50.039417+0000","last_active":"2023-11-24T06:18:50.041428+0000","last_peered":"2023-11-24T06:18:50.041428+0000","last_clean":"2023-11-24T06:18:50.041428+0000","last_became_active":"2023-11-24T06:18:50.031461+0000","last_became_peered":"2023-11-24T06:18:50.031461+0000","last_unstale":"2023-11-24T06:18:50.041428+0000","last_undegraded":"2023-11-24T06:18:50.041428+0000","last_fullsized":"2023-11-24T06:18:50.041428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036803+0000","last_change":"2023-11-24T06:18:49.015786+0000","last_active":"2023-11-24T06:18:50.036803+0000","last_peered":"2023-11-24T06:18:50.036803+0000","last_clean":"2023-11-24T06:18:50.036803+0000","last_became_active":"2023-11-24T06:18:49.015506+0000","last_became_peered":"2023-11-24T06:18:49.015506+0000","last_unstale":"2023-11-24T06:18:50.036803+0000","last_undegraded":"2023-11-24T06:18:50.036803+0000","last_fullsized":"2023-11-24T06:18:50.036803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:01.167887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024358+0000","last_change":"2023-11-24T06:18:50.023402+0000","last_active":"2023-11-24T06:18:51.024358+0000","last_peered":"2023-11-24T06:18:51.024358+0000","last_clean":"2023-11-24T06:18:51.024358+0000","last_became_active":"2023-11-24T06:18:50.023117+0000","last_became_peered":"2023-11-24T06:18:50.023117+0000","last_unstale":"2023-11-24T06:18:51.024358+0000","last_undegraded":"2023-11-24T06:18:51.024358+0000","last_fullsized":"2023-11-24T06:18:51.024358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:54.022789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.034041+0000","last_change":"2023-11-24T06:18:50.032212+0000","last_active":"2023-11-24T06:18:50.034041+0000","last_peered":"2023-11-24T06:18:50.034041+0000","last_clean":"2023-11-24T06:18:50.034041+0000","last_became_active":"2023-11-24T06:18:50.031933+0000","last_became_peered":"2023-11-24T06:18:50.031933+0000","last_unstale":"2023-11-24T06:18:50.034041+0000","last_undegraded":"2023-11-24T06:18:50.034041+0000","last_fullsized":"2023-11-24T06:18:50.034041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.038309+0000","last_change":"2023-11-24T06:18:50.037725+0000","last_active":"2023-11-24T06:18:50.038309+0000","last_peered":"2023-11-24T06:18:50.038309+0000","last_clean":"2023-11-24T06:18:50.038309+0000","last_became_active":"2023-11-24T06:18:50.037471+0000","last_became_peered":"2023-11-24T06:18:50.037471+0000","last_unstale":"2023-11-24T06:18:50.038309+0000","last_undegraded":"2023-11-24T06:18:50.038309+0000","last_fullsized":"2023-11-24T06:18:50.038309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041406+0000","last_change":"2023-11-24T06:18:50.038973+0000","last_active":"2023-11-24T06:18:50.041406+0000","last_peered":"2023-11-24T06:18:50.041406+0000","last_clean":"2023-11-24T06:18:50.041406+0000","last_became_active":"2023-11-24T06:18:50.038345+0000","last_became_peered":"2023-11-24T06:18:50.038345+0000","last_unstale":"2023-11-24T06:18:50.041406+0000","last_undegraded":"2023-11-24T06:18:50.041406+0000","last_fullsized":"2023-11-24T06:18:50.041406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:25.578418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.032822+0000","last_change":"2023-11-24T06:18:50.024376+0000","last_active":"2023-11-24T06:18:50.032822+0000","last_peered":"2023-11-24T06:18:50.032822+0000","last_clean":"2023-11-24T06:18:50.032822+0000","last_became_active":"2023-11-24T06:18:50.024118+0000","last_became_peered":"2023-11-24T06:18:50.024118+0000","last_unstale":"2023-11-24T06:18:50.032822+0000","last_undegraded":"2023-11-24T06:18:50.032822+0000","last_fullsized":"2023-11-24T06:18:50.032822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:47:17.509901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.036418+0000","last_change":"2023-11-24T06:18:50.028334+0000","last_active":"2023-11-24T06:18:50.036418+0000","last_peered":"2023-11-24T06:18:50.036418+0000","last_clean":"2023-11-24T06:18:50.036418+0000","last_became_active":"2023-11-24T06:18:50.028091+0000","last_became_peered":"2023-11-24T06:18:50.028091+0000","last_unstale":"2023-11-24T06:18:50.036418+0000","last_undegraded":"2023-11-24T06:18:50.036418+0000","last_fullsized":"2023-11-24T06:18:50.036418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021380+0000","last_change":"2023-11-24T06:18:50.025563+0000","last_active":"2023-11-24T06:18:51.021380+0000","last_peered":"2023-11-24T06:18:51.021380+0000","last_clean":"2023-11-24T06:18:51.021380+0000","last_became_active":"2023-11-24T06:18:50.025276+0000","last_became_peered":"2023-11-24T06:18:50.025276+0000","last_unstale":"2023-11-24T06:18:51.021380+0000","last_undegraded":"2023-11-24T06:18:51.021380+0000","last_fullsized":"2023-11-24T06:18:51.021380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.034911+0000","last_change":"2023-11-24T06:18:50.032672+0000","last_active":"2023-11-24T06:18:50.034911+0000","last_peered":"2023-11-24T06:18:50.034911+0000","last_clean":"2023-11-24T06:18:50.034911+0000","last_became_active":"2023-11-24T06:18:50.032400+0000","last_became_peered":"2023-11-24T06:18:50.032400+0000","last_unstale":"2023-11-24T06:18:50.034911+0000","last_undegraded":"2023-11-24T06:18:50.034911+0000","last_fullsized":"2023-11-24T06:18:50.034911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035709+0000","last_change":"2023-11-24T06:18:50.027189+0000","last_active":"2023-11-24T06:18:50.035709+0000","last_peered":"2023-11-24T06:18:50.035709+0000","last_clean":"2023-11-24T06:18:50.035709+0000","last_became_active":"2023-11-24T06:18:50.026893+0000","last_became_peered":"2023-11-24T06:18:50.026893+0000","last_unstale":"2023-11-24T06:18:50.035709+0000","last_undegraded":"2023-11-24T06:18:50.035709+0000","last_fullsized":"2023-11-24T06:18:50.035709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:21.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035986+0000","last_change":"2023-11-24T06:18:50.031274+0000","last_active":"2023-11-24T06:18:50.035986+0000","last_peered":"2023-11-24T06:18:50.035986+0000","last_clean":"2023-11-24T06:18:50.035986+0000","last_became_active":"2023-11-24T06:18:50.030997+0000","last_became_peered":"2023-11-24T06:18:50.030997+0000","last_unstale":"2023-11-24T06:18:50.035986+0000","last_undegraded":"2023-11-24T06:18:50.035986+0000","last_fullsized":"2023-11-24T06:18:50.035986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.035754+0000","last_change":"2023-11-24T06:18:50.027675+0000","last_active":"2023-11-24T06:18:50.035754+0000","last_peered":"2023-11-24T06:18:50.035754+0000","last_clean":"2023-11-24T06:18:50.035754+0000","last_became_active":"2023-11-24T06:18:50.027402+0000","last_became_peered":"2023-11-24T06:18:50.027402+0000","last_unstale":"2023-11-24T06:18:50.035754+0000","last_undegraded":"2023-11-24T06:18:50.035754+0000","last_fullsized":"2023-11-24T06:18:50.035754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:03.868748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.041439+0000","last_change":"2023-11-24T06:18:50.039174+0000","last_active":"2023-11-24T06:18:50.041439+0000","last_peered":"2023-11-24T06:18:50.041439+0000","last_clean":"2023-11-24T06:18:50.041439+0000","last_became_active":"2023-11-24T06:18:50.038671+0000","last_became_peered":"2023-11-24T06:18:50.038671+0000","last_unstale":"2023-11-24T06:18:50.041439+0000","last_undegraded":"2023-11-24T06:18:50.041439+0000","last_fullsized":"2023-11-24T06:18:50.041439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:03.614632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.034395+0000","last_change":"2023-11-24T06:18:50.032515+0000","last_active":"2023-11-24T06:18:50.034395+0000","last_peered":"2023-11-24T06:18:50.034395+0000","last_clean":"2023-11-24T06:18:50.034395+0000","last_became_active":"2023-11-24T06:18:50.032238+0000","last_became_peered":"2023-11-24T06:18:50.032238+0000","last_unstale":"2023-11-24T06:18:50.034395+0000","last_undegraded":"2023-11-24T06:18:50.034395+0000","last_fullsized":"2023-11-24T06:18:50.034395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:59:36.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.039901+0000","last_change":"2023-11-24T06:18:50.035460+0000","last_active":"2023-11-24T06:18:50.039901+0000","last_peered":"2023-11-24T06:18:50.039901+0000","last_clean":"2023-11-24T06:18:50.039901+0000","last_became_active":"2023-11-24T06:18:50.031929+0000","last_became_peered":"2023-11-24T06:18:50.031929+0000","last_unstale":"2023-11-24T06:18:50.039901+0000","last_undegraded":"2023-11-24T06:18:50.039901+0000","last_fullsized":"2023-11-24T06:18:50.039901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":660287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9289999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1629999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":69371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:18:54.503 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T06:18:54.503 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-24T06:18:54.503 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-24T06:18:54.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-24T06:18:54.503 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-24T06:18:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.658+0000 7fb5e648c700 1 Processor -- start 2023-11-24T06:18:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.658+0000 7fb5e648c700 1 -- start start 2023-11-24T06:18:54.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5e648c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5e00d2240 0x7fb5e00b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5e648c700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00b1d40 con 0x7fb5e015e1b0 2023-11-24T06:18:54.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5e648c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00b28c0 con 0x7fb5e015ddd0 2023-11-24T06:18:54.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5e648c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00b1f90 con 0x7fb5e00d2240 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5e4a29700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb5e015e1b0 0x7fb5e006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38578/0 (socket says 172.21.15.89:38578) 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5e4a29700 1 -- 172.21.15.89:0/2055034563 learned_addr learned my addr 172.21.15.89:0/2055034563 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5df7fe700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5e00d2240 0x7fb5e00b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024531456 0 0) 0x7fb5e00b1d40 con 0x7fb5e015e1b0 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c80035b0 con 0x7fb5e015e1b0 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1128635022 0 0) 0x7fb5c80035b0 con 0x7fb5e015e1b0 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5e00d2240 msgr2=0x7fb5e00b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5e00d2240 0x7fb5e00b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 >> v1:172.21.15.123:6789/0 conn(0x7fb5e015ddd0 legacy=0x7fb5e00d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00b2270 con 0x7fb5e015e1b0 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fb5d80019f0 con 0x7fb5e015e1b0 2023-11-24T06:18:54.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 >> v1:172.21.15.89:6789/0 conn(0x7fb5e015e1b0 legacy=0x7fb5e006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c8003a30 0x7fb5c8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5c8004850 0x7fb5c8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5e00d2240 0x7fb5c80052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00b1f90 con 0x7fb5c8003a30 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c800bbd0 con 0x7fb5c8004850 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c800bd90 con 0x7fb5e00d2240 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5dffff700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c8003a30 0x7fb5c8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.659+0000 7fb5df7fe700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5c8004850 0x7fb5c8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5e4a29700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5e00d2240 0x7fb5c80052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5dffff700 1 -- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5e00d2240 msgr2=0x7fb5c80052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5dffff700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5e00d2240 0x7fb5c80052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5dffff700 1 -- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5c8004850 msgr2=0x7fb5c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5dffff700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5c8004850 0x7fb5c8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5dffff700 1 -- 172.21.15.89:0/2055034563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c80035b0 con 0x7fb5c8003a30 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5df7fe700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5c8004850 0x7fb5c8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5dffff700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c8003a30 0x7fb5c8003e50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00017c0 tx=0x7fb5c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef97295f9be19c00 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00052d0 con 0x7fb5c8003a30 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d0005430 con 0x7fb5c8003a30 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5deffd700 1 -- 172.21.15.89:0/2055034563 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00038c0 con 0x7fb5c8003a30 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5e648c700 1 -- 172.21.15.89:0/2055034563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c8003a30 msgr2=0x7fb5c8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5e648c700 1 --2- 172.21.15.89:0/2055034563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c8003a30 0x7fb5c8003e50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00017c0 tx=0x7fb5c8005880 comp rx=0 tx=0).stop 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5e4a29700 1 -- 172.21.15.89:0/2055034563 reap_dead start 2023-11-24T06:18:54.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5e648c700 1 -- 172.21.15.89:0/2055034563 shutdown_connections 2023-11-24T06:18:54.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5e648c700 1 -- 172.21.15.89:0/2055034563 >> 172.21.15.89:0/2055034563 conn(0x7fb5e00731a0 msgr2=0x7fb5e009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:54.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5e648c700 1 -- 172.21.15.89:0/2055034563 shutdown_connections 2023-11-24T06:18:54.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.660+0000 7fb5e648c700 1 -- 172.21.15.89:0/2055034563 wait complete. 2023-11-24T06:18:54.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.662+0000 7fb5e648c700 1 Processor -- start 2023-11-24T06:18:54.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.662+0000 7fb5e648c700 1 -- start start 2023-11-24T06:18:54.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e648c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5c8003a30 0x7fb5e009bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e648c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5e009f630 0x7fb5e009d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e648c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5e009daa0 0x7fb5e014d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e648c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e014c000 con 0x7fb5e009f630 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e648c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e009c750 con 0x7fb5e009daa0 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e648c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e014b380 con 0x7fb5c8003a30 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5e009f630 0x7fb5e009d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5e009f630 0x7fb5e009d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35084/0 (socket says 172.21.15.89:35084) 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 -- 172.21.15.89:0/3262177561 learned_addr learned my addr 172.21.15.89:0/3262177561 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5dffff700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5c8003a30 0x7fb5e009bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e4a29700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5e009daa0 0x7fb5e014d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 -- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5c8003a30 msgr2=0x7fb5e009bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5c8003a30 0x7fb5e009bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 -- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5e009daa0 msgr2=0x7fb5e014d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5e009daa0 0x7fb5e014d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 -- 172.21.15.89:0/3262177561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00b2ed0 con 0x7fb5e009f630 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e4a29700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5e009daa0 0x7fb5e014d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5df7fe700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5e009f630 0x7fb5e009d560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40118d0 tx=0x7fb5d4011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5dcff9700 1 -- 172.21.15.89:0/3262177561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4004220 con 0x7fb5e009f630 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5dcff9700 1 -- 172.21.15.89:0/3262177561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d4004380 con 0x7fb5e009f630 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e014b660 con 0x7fb5e009f630 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5dcff9700 1 -- 172.21.15.89:0/3262177561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40036a0 con 0x7fb5e009f630 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.663+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e014bc20 con 0x7fb5e009f630 2023-11-24T06:18:54.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.664+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e014c140 con 0x7fb5e009f630 2023-11-24T06:18:54.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.665+0000 7fb5dcff9700 1 -- 172.21.15.89:0/3262177561 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb5d4003800 con 0x7fb5e009f630 2023-11-24T06:18:54.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.665+0000 7fb5dcff9700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb5cc06aed0 0x7fb5cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:54.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.665+0000 7fb5dffff700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb5cc06aed0 0x7fb5cc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:54.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.665+0000 7fb5dcff9700 1 -- 172.21.15.89:0/3262177561 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5d4089960 con 0x7fb5e009f630 2023-11-24T06:18:54.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.665+0000 7fb5dffff700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb5cc06aed0 0x7fb5cc06d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0015040 tx=0x7fb5d0009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:54.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.670+0000 7fb5dcff9700 1 -- 172.21.15.89:0/3262177561 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5e014c140 con 0x7fb5e009f630 2023-11-24T06:18:54.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.919+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb5e014c140 con 0x7fb5e009f630 2023-11-24T06:18:54.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.922+0000 7fb5dcff9700 1 -- 172.21.15.89:0/3262177561 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171029 (secure 0 0 0) 0x7fb5e014c140 con 0x7fb5e009f630 2023-11-24T06:18:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:54.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.924+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb5cc06aed0 msgr2=0x7fb5cc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.924+0000 7fb5e648c700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb5cc06aed0 0x7fb5cc06d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0015040 tx=0x7fb5d0009030 comp rx=0 tx=0).stop 2023-11-24T06:18:54.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.924+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5e009f630 msgr2=0x7fb5e009d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.924+0000 7fb5e648c700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5e009f630 0x7fb5e009d560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40118d0 tx=0x7fb5d4011c90 comp rx=0 tx=0).stop 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.925+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 shutdown_connections 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.925+0000 7fb5e648c700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb5cc06aed0 0x7fb5cc06d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.925+0000 7fb5e648c700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5e009daa0 0x7fb5e014d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.925+0000 7fb5e648c700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5c8003a30 0x7fb5e009bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.925+0000 7fb5e648c700 1 --2- 172.21.15.89:0/3262177561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5e009f630 0x7fb5e009d560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.925+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 >> 172.21.15.89:0/3262177561 conn(0x7fb5e00731a0 msgr2=0x7fb5e00d1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.925+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 shutdown_connections 2023-11-24T06:18:54.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:54.925+0000 7fb5e648c700 1 -- 172.21.15.89:0/3262177561 wait complete. 2023-11-24T06:18:54.938 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2291342889},{"type":"v1","addr":"172.21.15.89:6833","nonce":2291342889}]},"active_addr":"172.21.15.89:6833/2291342889","active_change":"2023-11-24T06:18:29.709498+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2488488813}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":559077458}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":874859423}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":476635603}]}]} 2023-11-24T06:18:54.941 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-24T06:18:54.942 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-24T06:18:54.942 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:55.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.059+0000 7f50fcf83700 1 Processor -- start 2023-11-24T06:18:55.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.059+0000 7f50fcf83700 1 -- start start 2023-11-24T06:18:55.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.060+0000 7f50fcf83700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f50f815bdd0 0x7f50f81604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.060+0000 7f50fcf83700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f80b22d0 con 0x7f50f80d1c70 2023-11-24T06:18:55.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.060+0000 7f50fcf83700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f80b25b0 con 0x7f50f80b3370 2023-11-24T06:18:55.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.060+0000 7f50fcf83700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f80b2f50 con 0x7f50f815bdd0 2023-11-24T06:18:55.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f659c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f50f815bdd0 0x7f50f81604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f6d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f50f80d1c70 0x7f50f8165ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38594/0 (socket says 172.21.15.89:38594) 2023-11-24T06:18:55.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f659c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f50f815bdd0 0x7f50f81604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35094/0 (socket says 172.21.15.89:35094) 2023-11-24T06:18:55.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f6d9d700 1 -- 172.21.15.89:0/1499694700 learned_addr learned my addr 172.21.15.89:0/1499694700 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:55.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f659c700 1 -- 172.21.15.89:0/1499694700 >> v1:172.21.15.123:6789/0 conn(0x7f50f80b3370 legacy=0x7f50f80b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f659c700 1 -- 172.21.15.89:0/1499694700 >> v1:172.21.15.89:6789/0 conn(0x7f50f80d1c70 legacy=0x7f50f8165ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f659c700 1 -- 172.21.15.89:0/1499694700 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f80b2890 con 0x7f50f815bdd0 2023-11-24T06:18:55.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f559a700 1 -- 172.21.15.89:0/1499694700 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645411793 0 0) 0x7f50f80b22d0 con 0x7f50f80d1c70 2023-11-24T06:18:55.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f659c700 1 --2- 172.21.15.89:0/1499694700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f81604e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009840 tx=0x7f50e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc26c0d4d7e838e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f559a700 1 -- 172.21.15.89:0/1499694700 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e0011830 con 0x7f50f815bdd0 2023-11-24T06:18:55.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f559a700 1 -- 172.21.15.89:0/1499694700 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e0011990 con 0x7f50f815bdd0 2023-11-24T06:18:55.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.061+0000 7f50f559a700 1 -- 172.21.15.89:0/1499694700 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e0011d20 con 0x7f50f815bdd0 2023-11-24T06:18:55.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.062+0000 7f50fcf83700 1 -- 172.21.15.89:0/1499694700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 msgr2=0x7f50f81604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.062+0000 7f50fcf83700 1 --2- 172.21.15.89:0/1499694700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f81604e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009840 tx=0x7f50e0009b50 comp rx=0 tx=0).stop 2023-11-24T06:18:55.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.062+0000 7f50fcf83700 1 -- 172.21.15.89:0/1499694700 shutdown_connections 2023-11-24T06:18:55.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.062+0000 7f50fcf83700 1 --2- 172.21.15.89:0/1499694700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f81604e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.062+0000 7f50fcf83700 1 -- 172.21.15.89:0/1499694700 >> 172.21.15.89:0/1499694700 conn(0x7f50f80731a0 msgr2=0x7f50f809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.062+0000 7f50fcf83700 1 -- 172.21.15.89:0/1499694700 shutdown_connections 2023-11-24T06:18:55.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.062+0000 7f50fcf83700 1 -- 172.21.15.89:0/1499694700 wait complete. 2023-11-24T06:18:55.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.063+0000 7f50fcf83700 1 Processor -- start 2023-11-24T06:18:55.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.063+0000 7f50fcf83700 1 -- start start 2023-11-24T06:18:55.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.064+0000 7f50fcf83700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50f80b3370 0x7f50f80b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.064+0000 7f50fcf83700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f80d1c70 0x7f50f8156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.064+0000 7f50fcf83700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f80b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.064+0000 7f50fcf83700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80cade0 con 0x7f50f815bdd0 2023-11-24T06:18:55.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.064+0000 7f50fcf83700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80b7330 con 0x7f50f80d1c70 2023-11-24T06:18:55.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.064+0000 7f50fcf83700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80b75f0 con 0x7f50f80b3370 2023-11-24T06:18:55.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.064+0000 7f50f6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f80b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f80b53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35100/0 (socket says 172.21.15.89:35100) 2023-11-24T06:18:55.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f6d9d700 1 -- 172.21.15.89:0/975068752 learned_addr learned my addr 172.21.15.89:0/975068752 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f5d9b700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f80d1c70 0x7f50f8156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f659c700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50f80b3370 0x7f50f80b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f6d9d700 1 -- 172.21.15.89:0/975068752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50f80b3370 msgr2=0x7f50f80b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f6d9d700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50f80b3370 0x7f50f80b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f6d9d700 1 -- 172.21.15.89:0/975068752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f80d1c70 msgr2=0x7f50f8156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f6d9d700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f80d1c70 0x7f50f8156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f6d9d700 1 -- 172.21.15.89:0/975068752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e00094a0 con 0x7f50f815bdd0 2023-11-24T06:18:55.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50f6d9d700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f80b53b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f50f0002230 tx=0x7f50f000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50eeffd700 1 -- 172.21.15.89:0/975068752 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f000aac0 con 0x7f50f815bdd0 2023-11-24T06:18:55.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50eeffd700 1 -- 172.21.15.89:0/975068752 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f000ac40 con 0x7f50f815bdd0 2023-11-24T06:18:55.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f8156f70 con 0x7f50f815bdd0 2023-11-24T06:18:55.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50eeffd700 1 -- 172.21.15.89:0/975068752 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f00070d0 con 0x7f50f815bdd0 2023-11-24T06:18:55.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.065+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f80ca720 con 0x7f50f815bdd0 2023-11-24T06:18:55.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.066+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f80caf20 con 0x7f50f815bdd0 2023-11-24T06:18:55.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.067+0000 7f50eeffd700 1 -- 172.21.15.89:0/975068752 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f50f0007250 con 0x7f50f815bdd0 2023-11-24T06:18:55.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.067+0000 7f50eeffd700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f50d806aed0 0x7f50d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.067+0000 7f50eeffd700 1 -- 172.21.15.89:0/975068752 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50f00893c0 con 0x7f50f815bdd0 2023-11-24T06:18:55.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.067+0000 7f50f659c700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f50d806aed0 0x7f50d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.067+0000 7f50f659c700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f50d806aed0 0x7f50d806d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f50e00090b0 tx=0x7f50e001c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.071+0000 7f50eeffd700 1 -- 172.21.15.89:0/975068752 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50f80caf20 con 0x7f50f815bdd0 2023-11-24T06:18:55.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.262+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50f80caf20 con 0x7f50f815bdd0 2023-11-24T06:18:55.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.263+0000 7f50eeffd700 1 -- 172.21.15.89:0/975068752 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f50f80caf20 con 0x7f50f815bdd0 2023-11-24T06:18:55.265 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:55.265 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:55.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f50d806aed0 msgr2=0x7f50d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f50d806aed0 0x7f50d806d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f50e00090b0 tx=0x7f50e001c040 comp rx=0 tx=0).stop 2023-11-24T06:18:55.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 msgr2=0x7f50f80b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f80b53b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f50f0002230 tx=0x7f50f000cf50 comp rx=0 tx=0).stop 2023-11-24T06:18:55.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 shutdown_connections 2023-11-24T06:18:55.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50f815bdd0 0x7f50f80b53b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50f80b3370 0x7f50f80b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f50d806aed0 0x7f50d806d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 --2- 172.21.15.89:0/975068752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f80d1c70 0x7f50f8156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 >> 172.21.15.89:0/975068752 conn(0x7f50f80731a0 msgr2=0x7f50f80b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 shutdown_connections 2023-11-24T06:18:55.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.265+0000 7f50fcf83700 1 -- 172.21.15.89:0/975068752 wait complete. 2023-11-24T06:18:55.279 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-24T06:18:55.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:55.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.397+0000 7fb199421700 1 Processor -- start 2023-11-24T06:18:55.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.397+0000 7fb199421700 1 -- start start 2023-11-24T06:18:55.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.397+0000 7fb199421700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b0fb0 0x7fb1940cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.397+0000 7fb199421700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1940d0280 0x7fb19415d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb199421700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1940b1c60 con 0x7fb1940b1390 2023-11-24T06:18:55.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb199421700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1940d0720 con 0x7fb1940d0280 2023-11-24T06:18:55.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb199421700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940d0a00 con 0x7fb1940b0fb0 2023-11-24T06:18:55.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb1937fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1940d0280 0x7fb19415d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb1937fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1940d0280 0x7fb19415d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35108/0 (socket says 172.21.15.89:35108) 2023-11-24T06:18:55.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb1937fe700 1 -- 172.21.15.89:0/3233559718 learned_addr learned my addr 172.21.15.89:0/3233559718 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb192ffd700 1 --2- 172.21.15.89:0/3233559718 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b0fb0 0x7fb1940cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb1937fe700 1 -- 172.21.15.89:0/3233559718 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b0fb0 msgr2=0x7fb1940cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb1937fe700 1 --2- 172.21.15.89:0/3233559718 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b0fb0 0x7fb1940cfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb1937fe700 1 -- 172.21.15.89:0/3233559718 >> v1:172.21.15.123:6789/0 conn(0x7fb1940b1390 legacy=0x7fb19406b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb1937fe700 1 -- 172.21.15.89:0/3233559718 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1940d0ce0 con 0x7fb1940d0280 2023-11-24T06:18:55.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb1937fe700 1 --2- 172.21.15.89:0/3233559718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940d0280 0x7fb19415d920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009e50 tx=0x7fb18c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419e405cf2af3a24 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb191ffb700 1 -- 172.21.15.89:0/3233559718 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c00e070 con 0x7fb1940d0280 2023-11-24T06:18:55.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb191ffb700 1 -- 172.21.15.89:0/3233559718 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18c00b910 con 0x7fb1940d0280 2023-11-24T06:18:55.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.398+0000 7fb191ffb700 1 -- 172.21.15.89:0/3233559718 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c00bcc0 con 0x7fb1940d0280 2023-11-24T06:18:55.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.399+0000 7fb199421700 1 -- 172.21.15.89:0/3233559718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940d0280 msgr2=0x7fb19415d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.399+0000 7fb199421700 1 --2- 172.21.15.89:0/3233559718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940d0280 0x7fb19415d920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009e50 tx=0x7fb18c00b330 comp rx=0 tx=0).stop 2023-11-24T06:18:55.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.399+0000 7fb199421700 1 -- 172.21.15.89:0/3233559718 shutdown_connections 2023-11-24T06:18:55.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.399+0000 7fb199421700 1 --2- 172.21.15.89:0/3233559718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940d0280 0x7fb19415d920 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.399+0000 7fb199421700 1 --2- 172.21.15.89:0/3233559718 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b0fb0 0x7fb1940cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.399+0000 7fb199421700 1 -- 172.21.15.89:0/3233559718 >> 172.21.15.89:0/3233559718 conn(0x7fb19409aa00 msgr2=0x7fb19409ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.399+0000 7fb199421700 1 -- 172.21.15.89:0/3233559718 shutdown_connections 2023-11-24T06:18:55.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.399+0000 7fb199421700 1 -- 172.21.15.89:0/3233559718 wait complete. 2023-11-24T06:18:55.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.400+0000 7fb199421700 1 Processor -- start 2023-11-24T06:18:55.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.400+0000 7fb199421700 1 -- start start 2023-11-24T06:18:55.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb199421700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940b0fb0 0x7fb1940c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb199421700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b1390 0x7fb1940b4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb199421700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1940d0280 0x7fb1940ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb199421700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940c99f0 con 0x7fb1940b0fb0 2023-11-24T06:18:55.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb199421700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940b4dd0 con 0x7fb1940d0280 2023-11-24T06:18:55.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb199421700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940b7410 con 0x7fb1940b1390 2023-11-24T06:18:55.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940b0fb0 0x7fb1940c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940b0fb0 0x7fb1940c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35110/0 (socket says 172.21.15.89:35110) 2023-11-24T06:18:55.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 -- 172.21.15.89:0/3155696781 learned_addr learned my addr 172.21.15.89:0/3155696781 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb1937fe700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1940d0280 0x7fb1940ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb1927fc700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b1390 0x7fb1940b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 -- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b1390 msgr2=0x7fb1940b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b1390 0x7fb1940b4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 -- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1940d0280 msgr2=0x7fb1940ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1940d0280 0x7fb1940ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 -- 172.21.15.89:0/3155696781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c009b30 con 0x7fb1940b0fb0 2023-11-24T06:18:55.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb1937fe700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1940d0280 0x7fb1940ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:55.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.401+0000 7fb192ffd700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940b0fb0 0x7fb1940c8190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb18400e9b0 tx=0x7fb18400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.402+0000 7fb17b7fe700 1 -- 172.21.15.89:0/3155696781 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb184009710 con 0x7fb1940b0fb0 2023-11-24T06:18:55.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.402+0000 7fb17b7fe700 1 -- 172.21.15.89:0/3155696781 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18400c850 con 0x7fb1940b0fb0 2023-11-24T06:18:55.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.402+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1940b76f0 con 0x7fb1940b0fb0 2023-11-24T06:18:55.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.402+0000 7fb17b7fe700 1 -- 172.21.15.89:0/3155696781 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18400cbc0 con 0x7fb1940b0fb0 2023-11-24T06:18:55.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.402+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1940c9f20 con 0x7fb1940b0fb0 2023-11-24T06:18:55.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.403+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb194068480 con 0x7fb1940b0fb0 2023-11-24T06:18:55.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.403+0000 7fb17b7fe700 1 -- 172.21.15.89:0/3155696781 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb18400cd20 con 0x7fb1940b0fb0 2023-11-24T06:18:55.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.403+0000 7fb17b7fe700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18006aed0 0x7fb18006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.403+0000 7fb17b7fe700 1 -- 172.21.15.89:0/3155696781 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1840897c0 con 0x7fb1940b0fb0 2023-11-24T06:18:55.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.403+0000 7fb1927fc700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18006aed0 0x7fb18006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.404+0000 7fb1927fc700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18006aed0 0x7fb18006d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb1940c8800 tx=0x7fb188009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.408+0000 7fb17b7fe700 1 -- 172.21.15.89:0/3155696781 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1840547e0 con 0x7fb1940b0fb0 2023-11-24T06:18:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.596+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1940cbfe0 con 0x7fb1940b0fb0 2023-11-24T06:18:55.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.598+0000 7fb17b7fe700 1 -- 172.21.15.89:0/3155696781 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7fb184021090 con 0x7fb1940b0fb0 2023-11-24T06:18:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:55.600 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.599+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18006aed0 msgr2=0x7fb18006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.599+0000 7fb199421700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18006aed0 0x7fb18006d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb1940c8800 tx=0x7fb188009040 comp rx=0 tx=0).stop 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.599+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940b0fb0 msgr2=0x7fb1940c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940b0fb0 0x7fb1940c8190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb18400e9b0 tx=0x7fb18400ed70 comp rx=0 tx=0).stop 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 shutdown_connections 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18006aed0 0x7fb18006d390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1940d0280 0x7fb1940ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1940b0fb0 0x7fb1940c8190 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 --2- 172.21.15.89:0/3155696781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1940b1390 0x7fb1940b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 >> 172.21.15.89:0/3155696781 conn(0x7fb19409aa00 msgr2=0x7fb1940b3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 shutdown_connections 2023-11-24T06:18:55.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.600+0000 7fb199421700 1 -- 172.21.15.89:0/3155696781 wait complete. 2023-11-24T06:18:55.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-24T06:18:55.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-24T06:18:55.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-24T06:18:55.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-24T06:18:55.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-24T06:18:55.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-24T06:18:55.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-24T06:18:55.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-24T06:18:55.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.822+0000 7fe33cd09700 1 Processor -- start 2023-11-24T06:18:55.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.822+0000 7fe33cd09700 1 -- start start 2023-11-24T06:18:55.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.823+0000 7fe33cd09700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe338162590 0x7fe33806ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.823+0000 7fe33cd09700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe3380ce220 0x7fe3380d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.823+0000 7fe33cd09700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3380cff40 con 0x7fe338073f90 2023-11-24T06:18:55.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.823+0000 7fe33cd09700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3380d0910 con 0x7fe338162590 2023-11-24T06:18:55.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.823+0000 7fe33cd09700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3380ce6b0 con 0x7fe3380ce220 2023-11-24T06:18:55.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe336d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe338073f90 0x7fe33806dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38624/0 (socket says 172.21.15.89:38624) 2023-11-24T06:18:55.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe336d9d700 1 -- 172.21.15.89:0/2217651836 learned_addr learned my addr 172.21.15.89:0/2217651836 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:55.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe335d9b700 1 --2- 172.21.15.89:0/2217651836 >> v2:172.21.15.123:3300/0 conn(0x7fe3380ce220 0x7fe3380d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe33559a700 1 -- 172.21.15.89:0/2217651836 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487622625 0 0) 0x7fe3380cff40 con 0x7fe338073f90 2023-11-24T06:18:55.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe33559a700 1 -- 172.21.15.89:0/2217651836 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3200035b0 con 0x7fe338073f90 2023-11-24T06:18:55.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe335d9b700 1 -- 172.21.15.89:0/2217651836 >> v2:172.21.15.89:3300/0 conn(0x7fe338162590 msgr2=0x7fe33806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe335d9b700 1 --2- 172.21.15.89:0/2217651836 >> v2:172.21.15.89:3300/0 conn(0x7fe338162590 0x7fe33806ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe335d9b700 1 -- 172.21.15.89:0/2217651836 >> v1:172.21.15.89:6789/0 conn(0x7fe338073f90 legacy=0x7fe33806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.824+0000 7fe335d9b700 1 -- 172.21.15.89:0/2217651836 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33806c100 con 0x7fe3380ce220 2023-11-24T06:18:55.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.825+0000 7f9b02053700 1 Processor -- start 2023-11-24T06:18:55.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.825+0000 7fe335d9b700 1 --2- 172.21.15.89:0/2217651836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3380ce220 0x7fe3380d03d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3380b0580 tx=0x7fe328009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88a634fe0d2ba322 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.825+0000 7f9b02053700 1 -- start start 2023-11-24T06:18:55.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.825+0000 7fe33559a700 1 -- 172.21.15.89:0/2217651836 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328011830 con 0x7fe3380ce220 2023-11-24T06:18:55.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.825+0000 7fe33559a700 1 -- 172.21.15.89:0/2217651836 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe328011990 con 0x7fe3380ce220 2023-11-24T06:18:55.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.825+0000 7fe33559a700 1 -- 172.21.15.89:0/2217651836 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328011d00 con 0x7fe3380ce220 2023-11-24T06:18:55.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7fe33cd09700 1 -- 172.21.15.89:0/2217651836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3380ce220 msgr2=0x7fe3380d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7fe33cd09700 1 --2- 172.21.15.89:0/2217651836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3380ce220 0x7fe3380d03d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3380b0580 tx=0x7fe328009b20 comp rx=0 tx=0).stop 2023-11-24T06:18:55.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9b02053700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9afc0b4260 0x7f9afc0714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9b02053700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 0x7f9afc140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9b02053700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9afc071e90 con 0x7f9afc15d6a0 2023-11-24T06:18:55.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9b02053700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9afc072d60 con 0x7f9afc0b4260 2023-11-24T06:18:55.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9b02053700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc06de80 con 0x7f9afc071a10 2023-11-24T06:18:55.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7fe33cd09700 1 -- 172.21.15.89:0/2217651836 shutdown_connections 2023-11-24T06:18:55.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7fe33cd09700 1 --2- 172.21.15.89:0/2217651836 >> v2:172.21.15.89:3300/0 conn(0x7fe338162590 0x7fe33806ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7fe33cd09700 1 --2- 172.21.15.89:0/2217651836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3380ce220 0x7fe3380d03d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7fe33cd09700 1 -- 172.21.15.89:0/2217651836 >> 172.21.15.89:0/2217651836 conn(0x7fe3380cd570 msgr2=0x7fe3380cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9afbfff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 0x7f9afc140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9afbfff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 0x7f9afc140fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44000/0 (socket says 172.21.15.89:44000) 2023-11-24T06:18:55.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9afbfff700 1 -- 172.21.15.89:0/2317004888 learned_addr learned my addr 172.21.15.89:0/2317004888 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7f9afaffd700 1 --2- 172.21.15.89:0/2317004888 >> v2:172.21.15.123:3300/0 conn(0x7f9afc0b4260 0x7f9afc0714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7fe33cd09700 1 -- 172.21.15.89:0/2217651836 shutdown_connections 2023-11-24T06:18:55.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.826+0000 7fe33cd09700 1 -- 172.21.15.89:0/2217651836 wait complete. 2023-11-24T06:18:55.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f9afbfff700 1 -- 172.21.15.89:0/2317004888 >> v2:172.21.15.123:3300/0 conn(0x7f9afc0b4260 msgr2=0x7f9afc0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f9afbfff700 1 --2- 172.21.15.89:0/2317004888 >> v2:172.21.15.123:3300/0 conn(0x7f9afc0b4260 0x7f9afc0714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f9afbfff700 1 -- 172.21.15.89:0/2317004888 >> v1:172.21.15.123:6789/0 conn(0x7f9afc15d6a0 legacy=0x7f9afc0b3b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:55.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f9afbfff700 1 -- 172.21.15.89:0/2317004888 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc0720e0 con 0x7f9afc071a10 2023-11-24T06:18:55.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f9afbfff700 1 --2- 172.21.15.89:0/2317004888 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 0x7f9afc140fd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0088f0 tx=0x7f9aec00d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88c7aacab61f55e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f9afa7fc700 1 -- 172.21.15.89:0/2317004888 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec010070 con 0x7f9afc071a10 2023-11-24T06:18:55.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f9afa7fc700 1 -- 172.21.15.89:0/2317004888 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec00d990 con 0x7f9afc071a10 2023-11-24T06:18:55.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f9afa7fc700 1 -- 172.21.15.89:0/2317004888 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00dd20 con 0x7f9afc071a10 2023-11-24T06:18:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f68c0ed7700 1 Processor -- start 2023-11-24T06:18:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f68c0ed7700 1 -- start start 2023-11-24T06:18:55.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.827+0000 7f6d1eacf700 1 Processor -- start 2023-11-24T06:18:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f9b02053700 1 -- 172.21.15.89:0/2317004888 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 msgr2=0x7f9afc140fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f9b02053700 1 --2- 172.21.15.89:0/2317004888 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 0x7f9afc140fd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0088f0 tx=0x7f9aec00d330 comp rx=0 tx=0).stop 2023-11-24T06:18:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f9b02053700 1 -- 172.21.15.89:0/2317004888 shutdown_connections 2023-11-24T06:18:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f9b02053700 1 --2- 172.21.15.89:0/2317004888 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 0x7f9afc140fd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f9b02053700 1 --2- 172.21.15.89:0/2317004888 >> v2:172.21.15.123:3300/0 conn(0x7f9afc0b4260 0x7f9afc0714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f9b02053700 1 -- 172.21.15.89:0/2317004888 >> 172.21.15.89:0/2317004888 conn(0x7f9afc0cfa30 msgr2=0x7f9afc0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f68c0ed7700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f68bc164a90 0x7f68bc06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f68c0ed7700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68bc06ced0 con 0x7f68bc0b3b50 2023-11-24T06:18:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f68c0ed7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68bc06da70 con 0x7f68bc06d4d0 2023-11-24T06:18:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f68c0ed7700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68bc1634e0 con 0x7f68bc164a90 2023-11-24T06:18:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f68bad9d700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f68bc06d4d0 0x7f68bc16bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:42456/0 (socket says 172.21.15.89:42456) 2023-11-24T06:18:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f68bad9d700 1 -- 172.21.15.89:0/3213634332 learned_addr learned my addr 172.21.15.89:0/3213634332 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:55.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f68b9d9b700 1 --2- 172.21.15.89:0/3213634332 >> v2:172.21.15.123:3300/0 conn(0x7f68bc164a90 0x7f68bc06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7f6d1eacf700 1 -- start start 2023-11-24T06:18:55.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1eacf700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d180b56e0 0x7f6d18164f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1eacf700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180b3640 con 0x7f6d18159f60 2023-11-24T06:18:55.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1eacf700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180b3920 con 0x7f6d180b30d0 2023-11-24T06:18:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1eacf700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d180b3c00 con 0x7f6d180b56e0 2023-11-24T06:18:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7fe33cd09700 1 Processor -- start 2023-11-24T06:18:55.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.828+0000 7fe33cd09700 1 -- start start 2023-11-24T06:18:55.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f68b9d9b700 1 -- 172.21.15.89:0/3213634332 >> v1:172.21.15.123:6789/0 conn(0x7f68bc06d4d0 legacy=0x7f68bc16bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f68b9d9b700 1 -- 172.21.15.89:0/3213634332 >> v1:172.21.15.89:6789/0 conn(0x7f68bc0b3b50 legacy=0x7f68bc0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f68b9d9b700 1 -- 172.21.15.89:0/3213634332 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68bc1637c0 con 0x7f68bc164a90 2023-11-24T06:18:55.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f68b9d9b700 1 --2- 172.21.15.89:0/3213634332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68bc164a90 0x7f68bc06eee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009d90 tx=0x7f68b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68c7cdb9880917d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1d06c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d180b56e0 0x7f6d18164f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1d06c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d180b56e0 0x7f6d18164f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35136/0 (socket says 172.21.15.89:35136) 2023-11-24T06:18:55.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1d06c700 1 -- 172.21.15.89:0/3034018327 learned_addr learned my addr 172.21.15.89:0/3034018327 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7fe33cd09700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe338073f90 0x7fe338152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7fe33cd09700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3380ce220 0x7fe3381fb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7fe33cd09700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe338162590 0x7fe33815a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7fe33cd09700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3381531e0 con 0x7fe338073f90 2023-11-24T06:18:55.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7fe33cd09700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3381fc160 con 0x7fe338162590 2023-11-24T06:18:55.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7fe33cd09700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3381fc440 con 0x7fe3380ce220 2023-11-24T06:18:55.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1d06c700 1 -- 172.21.15.89:0/3034018327 >> v1:172.21.15.123:6789/0 conn(0x7f6d180b30d0 legacy=0x7f6d180b4fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:55.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1d06c700 1 -- 172.21.15.89:0/3034018327 >> v1:172.21.15.89:6789/0 conn(0x7f6d18159f60 legacy=0x7f6d1815a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:18:55.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.829+0000 7f6d1d06c700 1 -- 172.21.15.89:0/3034018327 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d180b3ee0 con 0x7f6d180b56e0 2023-11-24T06:18:55.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe338073f90 0x7fe338152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe338073f90 0x7fe338152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35140/0 (socket says 172.21.15.89:35140) 2023-11-24T06:18:55.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 -- 172.21.15.89:0/1649159492 learned_addr learned my addr 172.21.15.89:0/1649159492 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe336d9d700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe338162590 0x7fe33815a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7f6d1d06c700 1 --2- 172.21.15.89:0/3034018327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d180b56e0 0x7f6d18164f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000d370 tx=0x7f6d1000d680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befcad4b81ba8b6b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 -- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3380ce220 msgr2=0x7fe3381fb880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-24T06:18:55.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3380ce220 0x7fe3381fb880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 -- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe338162590 msgr2=0x7fe33815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe338162590 0x7fe33815a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 -- 172.21.15.89:0/1649159492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3280094a0 con 0x7fe338073f90 2023-11-24T06:18:55.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7f6d177fe700 1 -- 172.21.15.89:0/3034018327 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10010070 con 0x7f6d180b56e0 2023-11-24T06:18:55.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7f6d177fe700 1 -- 172.21.15.89:0/3034018327 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1000dc90 con 0x7f6d180b56e0 2023-11-24T06:18:55.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7f6d177fe700 1 -- 172.21.15.89:0/3034018327 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10007b30 con 0x7f6d180b56e0 2023-11-24T06:18:55.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.831+0000 7f6d1eacf700 1 -- 172.21.15.89:0/3034018327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d180b56e0 msgr2=0x7f6d18164f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.831+0000 7f6d1eacf700 1 --2- 172.21.15.89:0/3034018327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d180b56e0 0x7f6d18164f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000d370 tx=0x7f6d1000d680 comp rx=0 tx=0).stop 2023-11-24T06:18:55.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.831+0000 7f6d1eacf700 1 -- 172.21.15.89:0/3034018327 shutdown_connections 2023-11-24T06:18:55.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.831+0000 7f6d1eacf700 1 --2- 172.21.15.89:0/3034018327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d180b56e0 0x7f6d18164f50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.831+0000 7f6d1eacf700 1 -- 172.21.15.89:0/3034018327 >> 172.21.15.89:0/3034018327 conn(0x7f6d1806cec0 msgr2=0x7f6d1806d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe336d9d700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe338162590 0x7fe33815a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:55.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33659c700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe338073f90 0x7fe338152ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00b530 tx=0x7fe32c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c011830 con 0x7fe338073f90 2023-11-24T06:18:55.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32c0119b0 con 0x7fe338073f90 2023-11-24T06:18:55.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3381fc720 con 0x7fe338073f90 2023-11-24T06:18:55.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.830+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c011d60 con 0x7fe338073f90 2023-11-24T06:18:55.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.831+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3380c5dd0 con 0x7fe338073f90 2023-11-24T06:18:55.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f9b02053700 1 -- 172.21.15.89:0/2317004888 shutdown_connections 2023-11-24T06:18:55.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f68b959a700 1 -- 172.21.15.89:0/3213634332 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b400e070 con 0x7f68bc164a90 2023-11-24T06:18:55.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f68b959a700 1 -- 172.21.15.89:0/3213634332 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b400bb50 con 0x7f68bc164a90 2023-11-24T06:18:55.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f68b959a700 1 -- 172.21.15.89:0/3213634332 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4003a20 con 0x7f68bc164a90 2023-11-24T06:18:55.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe32c01a400 con 0x7fe338073f90 2023-11-24T06:18:55.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe318000fb0 con 0x7fe338073f90 2023-11-24T06:18:55.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f083da47700 1 Processor -- start 2023-11-24T06:18:55.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f6d1eacf700 1 -- 172.21.15.89:0/3034018327 shutdown_connections 2023-11-24T06:18:55.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f6d1eacf700 1 -- 172.21.15.89:0/3034018327 wait complete. 2023-11-24T06:18:55.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f083da47700 1 -- start start 2023-11-24T06:18:55.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f68c0ed7700 1 -- 172.21.15.89:0/3213634332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68bc164a90 msgr2=0x7f68bc06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.832+0000 7f68c0ed7700 1 --2- 172.21.15.89:0/3213634332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68bc164a90 0x7f68bc06eee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009d90 tx=0x7f68b400b330 comp rx=0 tx=0).stop 2023-11-24T06:18:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.833+0000 7fe326ffd700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe32006b330 0x7fe32006d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.833+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe32c089630 con 0x7fe338073f90 2023-11-24T06:18:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7fe335d9b700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe32006b330 0x7fe32006d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7fe326ffd700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7fe3200734d0 0x7fe3200758f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 --> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe320075fa0 con 0x7fe3200734d0 2023-11-24T06:18:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe32c08bcf0 con 0x7fe338073f90 2023-11-24T06:18:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f6d1eacf700 1 Processor -- start 2023-11-24T06:18:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7fe336d9d700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7fe3200734d0 0x7fe3200758f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7fe335d9b700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe32006b330 0x7fe32006d7f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3381fb7e0 tx=0x7fe328008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f083da47700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0838071be0 0x7f0838164bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f083da47700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08380720e0 con 0x7f083815d6a0 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f083da47700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08380723c0 con 0x7f08380b4260 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f083da47700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0838072d60 con 0x7f0838071be0 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f08377fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0838071be0 0x7f0838164bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f08377fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0838071be0 0x7f0838164bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50260/0 (socket says 172.21.15.89:50260) 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f08377fe700 1 -- 172.21.15.89:0/1118886679 learned_addr learned my addr 172.21.15.89:0/1118886679 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f08377fe700 1 -- 172.21.15.89:0/1118886679 >> v1:172.21.15.123:6789/0 conn(0x7f08380b4260 legacy=0x7f08380714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f08377fe700 1 -- 172.21.15.89:0/1118886679 >> v1:172.21.15.89:6789/0 conn(0x7f083815d6a0 legacy=0x7f08380b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f08377fe700 1 -- 172.21.15.89:0/1118886679 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08380726a0 con 0x7f0838071be0 2023-11-24T06:18:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7fe336d9d700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7fe3200734d0 0x7fe3200758f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f08377fe700 1 --2- 172.21.15.89:0/1118886679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0838071be0 0x7f0838164bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0830009e50 tx=0x7f083000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f370e69a24a7425c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f0835ffb700 1 -- 172.21.15.89:0/1118886679 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830003a20 con 0x7f0838071be0 2023-11-24T06:18:55.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f0835ffb700 1 -- 172.21.15.89:0/1118886679 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0830003b80 con 0x7f0838071be0 2023-11-24T06:18:55.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f0835ffb700 1 -- 172.21.15.89:0/1118886679 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083000b980 con 0x7f0838071be0 2023-11-24T06:18:55.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f9b02053700 1 -- 172.21.15.89:0/2317004888 wait complete. 2023-11-24T06:18:55.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f9b02053700 1 Processor -- start 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.834+0000 7f6d1eacf700 1 -- start start 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1eacf700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6d180b56e0 0x7f6d181fbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1eacf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d18159f60 0x7f6d1814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1eacf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d181e5be0 0x7f6d1815b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1eacf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181e3c30 con 0x7f6d181e5be0 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1eacf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18153800 con 0x7f6d18159f60 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1eacf700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181fccd0 con 0x7f6d180b56e0 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1d06c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d181e5be0 0x7f6d1815b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1d06c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d181e5be0 0x7f6d1815b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35158/0 (socket says 172.21.15.89:35158) 2023-11-24T06:18:55.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1d06c700 1 -- 172.21.15.89:0/2687908031 learned_addr learned my addr 172.21.15.89:0/2687908031 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d17fff700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d18159f60 0x7f6d1814fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.835+0000 7f6d1c86b700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6d180b56e0 0x7f6d181fbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d17fff700 1 -- 172.21.15.89:0/2687908031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6d180b56e0 msgr2=0x7f6d181fbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d17fff700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6d180b56e0 0x7f6d181fbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d17fff700 1 -- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d181e5be0 msgr2=0x7f6d1815b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d17fff700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d181e5be0 0x7f6d1815b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d17fff700 1 -- 172.21.15.89:0/2687908031 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1000d050 con 0x7f6d18159f60 2023-11-24T06:18:55.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f9b02053700 1 -- start start 2023-11-24T06:18:55.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 <== osd.4 v2:172.21.15.123:6816/1261947525 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe320075fa0 con 0x7fe3200734d0 2023-11-24T06:18:55.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d17fff700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d18159f60 0x7f6d1814fb10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00b450 tx=0x7f6d0c00b760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c011620 con 0x7f6d18159f60 2023-11-24T06:18:55.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f083da47700 1 -- 172.21.15.89:0/1118886679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0838071be0 msgr2=0x7f0838164bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f083da47700 1 --2- 172.21.15.89:0/1118886679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0838071be0 0x7f0838164bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0830009e50 tx=0x7f083000b330 comp rx=0 tx=0).stop 2023-11-24T06:18:55.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f083da47700 1 -- 172.21.15.89:0/1118886679 shutdown_connections 2023-11-24T06:18:55.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f083da47700 1 --2- 172.21.15.89:0/1118886679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0838071be0 0x7f0838164bb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f083da47700 1 -- 172.21.15.89:0/1118886679 >> 172.21.15.89:0/1118886679 conn(0x7f08380cfa30 msgr2=0x7f08380cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c0117a0 con 0x7f6d18159f60 2023-11-24T06:18:55.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c011b50 con 0x7f6d18159f60 2023-11-24T06:18:55.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d18150050 con 0x7f6d18159f60 2023-11-24T06:18:55.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.836+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d181fc710 con 0x7f6d18159f60 2023-11-24T06:18:55.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f68c0ed7700 1 -- 172.21.15.89:0/3213634332 shutdown_connections 2023-11-24T06:18:55.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f68c0ed7700 1 --2- 172.21.15.89:0/3213634332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68bc164a90 0x7f68bc06eee0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f68c0ed7700 1 -- 172.21.15.89:0/3213634332 >> 172.21.15.89:0/3213634332 conn(0x7f68bc0cfa30 msgr2=0x7f68bc0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f68c0ed7700 1 -- 172.21.15.89:0/3213634332 shutdown_connections 2023-11-24T06:18:55.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f68c0ed7700 1 -- 172.21.15.89:0/3213634332 wait complete. 2023-11-24T06:18:55.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f68c0ed7700 1 Processor -- start 2023-11-24T06:18:55.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6d0c011cb0 con 0x7f6d18159f60 2023-11-24T06:18:55.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6d180b54f0 con 0x7f6d18159f60 2023-11-24T06:18:55.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f6d157fa700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6d0406aed0 0x7f6d0406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d0c089000 con 0x7f6d18159f60 2023-11-24T06:18:55.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f6d157fa700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f6d04073070 0x7f6d04075490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 --> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d04075b40 con 0x7f6d04073070 2023-11-24T06:18:55.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d0c08b6c0 con 0x7f6d18159f60 2023-11-24T06:18:55.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f6d1d06c700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f6d04073070 0x7f6d04075490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f6d1c86b700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6d0406aed0 0x7f6d0406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7ff20d1f9700 1 Processor -- start 2023-11-24T06:18:55.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7ff20d1f9700 1 -- start start 2023-11-24T06:18:55.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff20d1f9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2080b3b50 0x7ff2080b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff20d1f9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff208164a90 0x7ff20816bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff20d1f9700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20806ced0 con 0x7ff20806d4d0 2023-11-24T06:18:55.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff20d1f9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20806da70 con 0x7ff208164a90 2023-11-24T06:18:55.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff20d1f9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b4470 con 0x7ff2080b3b50 2023-11-24T06:18:55.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f083da47700 1 -- 172.21.15.89:0/1118886679 shutdown_connections 2023-11-24T06:18:55.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9b02053700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 0x7f9afc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9b02053700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9afc0b4260 0x7f9afc0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9b02053700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9afc15d6a0 0x7f9afc140580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9b02053700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc15c240 con 0x7f9afc15d6a0 2023-11-24T06:18:55.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9b02053700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc0cf390 con 0x7f9afc0b4260 2023-11-24T06:18:55.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9b02053700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc15b910 con 0x7f9afc071a10 2023-11-24T06:18:55.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9afbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9afc15d6a0 0x7f9afc140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9afbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9afc15d6a0 0x7f9afc140580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35162/0 (socket says 172.21.15.89:35162) 2023-11-24T06:18:55.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.837+0000 7f9afbfff700 1 -- 172.21.15.89:0/3560293622 learned_addr learned my addr 172.21.15.89:0/3560293622 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f9afbfff700 1 -- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 msgr2=0x7f9afc0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f9afbfff700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9afc071a10 0x7f9afc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f9afbfff700 1 -- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9afc0b4260 msgr2=0x7f9afc0ceaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f9afbfff700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9afc0b4260 0x7f9afc0ceaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f9afbfff700 1 -- 172.21.15.89:0/3560293622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec0085d0 con 0x7f9afc15d6a0 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f9afbfff700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9afc15d6a0 0x7f9afc140580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9aec019040 tx=0x7f9aec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec008d90 con 0x7f9afc15d6a0 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec0036a0 con 0x7f9afc15d6a0 2023-11-24T06:18:55.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec003a50 con 0x7f9afc15d6a0 2023-11-24T06:18:55.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff20759e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff208164a90 0x7ff20816bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff20759e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff208164a90 0x7ff20816bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50282/0 (socket says 172.21.15.89:50282) 2023-11-24T06:18:55.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff20759e700 1 -- 172.21.15.89:0/17728058 learned_addr learned my addr 172.21.15.89:0/17728058 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.839+0000 7ff206d9d700 1 --2- 172.21.15.89:0/17728058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2080b3b50 0x7ff2080b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7f6d1d06c700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f6d04073070 0x7f6d04075490 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 <== osd.1 v2:172.21.15.89:6808/4291702656 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f6d04075b40 con 0x7f6d04073070 2023-11-24T06:18:55.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f6d1c86b700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6d0406aed0 0x7f6d0406d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6d181e55a0 tx=0x7f6d080090e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7ff20759e700 1 -- 172.21.15.89:0/17728058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2080b3b50 msgr2=0x7ff2080b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7ff20759e700 1 --2- 172.21.15.89:0/17728058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2080b3b50 0x7ff2080b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7ff20759e700 1 -- 172.21.15.89:0/17728058 >> v1:172.21.15.89:6789/0 conn(0x7ff20806d4d0 legacy=0x7ff20806eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7ff20759e700 1 -- 172.21.15.89:0/17728058 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2081634e0 con 0x7ff208164a90 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7ff20759e700 1 --2- 172.21.15.89:0/17728058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff208164a90 0x7ff20816bee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff200009e50 tx=0x7ff20000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec832f947f48832a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7ff205d9b700 1 -- 172.21.15.89:0/17728058 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000e070 con 0x7ff208164a90 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7ff205d9b700 1 -- 172.21.15.89:0/17728058 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff200003a20 con 0x7ff208164a90 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7ff205d9b700 1 -- 172.21.15.89:0/17728058 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200003dd0 con 0x7ff208164a90 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.838+0000 7f083da47700 1 -- 172.21.15.89:0/1118886679 wait complete. 2023-11-24T06:18:55.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7f083da47700 1 Processor -- start 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7f083da47700 1 -- start start 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f083da47700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0838071be0 0x7f08382038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f083da47700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380b4260 0x7f08380aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f083da47700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f083815d6a0 0x7f08380cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f083da47700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083815b780 con 0x7f08380b4260 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f083da47700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083815ac00 con 0x7f083815d6a0 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f083da47700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083815aee0 con 0x7f0838071be0 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f08377fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f083815d6a0 0x7f08380cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f08377fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f083815d6a0 0x7f08380cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50290/0 (socket says 172.21.15.89:50290) 2023-11-24T06:18:55.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.841+0000 7f08377fe700 1 -- 172.21.15.89:0/3929724132 learned_addr learned my addr 172.21.15.89:0/3929724132 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f08367fc700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380b4260 0x7f08380aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9afc15bbf0 con 0x7f9afc15d6a0 2023-11-24T06:18:55.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9afc0cefe0 con 0x7f9afc15d6a0 2023-11-24T06:18:55.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.840+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9afc0aedb0 con 0x7f9afc15d6a0 2023-11-24T06:18:55.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f08367fc700 1 -- 172.21.15.89:0/3929724132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0838071be0 msgr2=0x7f08382038c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:18:55.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f08367fc700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0838071be0 0x7f08382038c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f08367fc700 1 -- 172.21.15.89:0/3929724132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f083815d6a0 msgr2=0x7f08380cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f08367fc700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f083815d6a0 0x7f08380cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f08367fc700 1 -- 172.21.15.89:0/3929724132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0830009b30 con 0x7f08380b4260 2023-11-24T06:18:55.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7ff20d1f9700 1 -- 172.21.15.89:0/17728058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff208164a90 msgr2=0x7ff20816bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7ff20d1f9700 1 --2- 172.21.15.89:0/17728058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff208164a90 0x7ff20816bee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff200009e50 tx=0x7ff20000b330 comp rx=0 tx=0).stop 2023-11-24T06:18:55.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7ff20d1f9700 1 -- 172.21.15.89:0/17728058 shutdown_connections 2023-11-24T06:18:55.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7ff20d1f9700 1 --2- 172.21.15.89:0/17728058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2080b3b50 0x7ff2080b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7ff20d1f9700 1 --2- 172.21.15.89:0/17728058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff208164a90 0x7ff20816bee0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7ff20d1f9700 1 -- 172.21.15.89:0/17728058 >> 172.21.15.89:0/17728058 conn(0x7ff2080cfa30 msgr2=0x7ff2080cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f08367fc700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380b4260 0x7f08380aecb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f082c000f60 tx=0x7f082c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0052d0 con 0x7f08380b4260 2023-11-24T06:18:55.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c005430 con 0x7f08380b4260 2023-11-24T06:18:55.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0038a0 con 0x7f08380b4260 2023-11-24T06:18:55.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083815b1c0 con 0x7f08380b4260 2023-11-24T06:18:55.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0838203e00 con 0x7f08380b4260 2023-11-24T06:18:55.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9aec003bb0 con 0x7f9afc15d6a0 2023-11-24T06:18:55.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f9ae3fff700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9ae806afa0 0x7f9ae806d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9aec08a8a0 con 0x7f9afc15d6a0 2023-11-24T06:18:55.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f9ae3fff700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f9ae80731c0 0x7f9ae80755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 --> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ae8075c90 con 0x7f9ae80731c0 2023-11-24T06:18:55.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9aec08ac50 con 0x7f9afc15d6a0 2023-11-24T06:18:55.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0838071270 con 0x7f08380b4260 2023-11-24T06:18:55.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.842+0000 7f68c0ed7700 1 -- start start 2023-11-24T06:18:55.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68c0ed7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f68bc06d4d0 0x7f68bc14f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68c0ed7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68bc0b3b50 0x7f68bc0ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68c0ed7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc164a90 0x7f68bc0cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68c0ed7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc1536c0 con 0x7f68bc164a90 2023-11-24T06:18:55.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68c0ed7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc1553c0 con 0x7f68bc0b3b50 2023-11-24T06:18:55.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68c0ed7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc151f30 con 0x7f68bc06d4d0 2023-11-24T06:18:55.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68ba59c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f68bc06d4d0 0x7f68bc14f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68ba59c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f68bc06d4d0 0x7f68bc14f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44056/0 (socket says 172.21.15.89:44056) 2023-11-24T06:18:55.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.843+0000 7f68ba59c700 1 -- 172.21.15.89:0/4114446432 learned_addr learned my addr 172.21.15.89:0/4114446432 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7f68bad9d700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc164a90 0x7f68bc0cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7f68b9d9b700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68bc0b3b50 0x7f68bc0ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7ff20d1f9700 1 -- 172.21.15.89:0/17728058 shutdown_connections 2023-11-24T06:18:55.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7f68ba59c700 1 -- 172.21.15.89:0/4114446432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68bc0b3b50 msgr2=0x7f68bc0ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7f68ba59c700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68bc0b3b50 0x7f68bc0ceb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7f68ba59c700 1 -- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc164a90 msgr2=0x7f68bc0cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7f68ba59c700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc164a90 0x7f68bc0cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7f68ba59c700 1 -- 172.21.15.89:0/4114446432 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b4009a70 con 0x7f68bc06d4d0 2023-11-24T06:18:55.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.844+0000 7ff20d1f9700 1 -- 172.21.15.89:0/17728058 wait complete. 2023-11-24T06:18:55.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.845+0000 7f9afaffd700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f9ae80731c0 0x7f9ae80755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.845+0000 7f9afb7fe700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9ae806afa0 0x7f9ae806d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7ff20d1f9700 1 Processor -- start 2023-11-24T06:18:55.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7ff20d1f9700 1 -- start start 2023-11-24T06:18:55.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7f9afaffd700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f9ae80731c0 0x7f9ae80755e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7f9afb7fe700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9ae806afa0 0x7f9ae806d460 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9af00096d0 tx=0x7f9af0006c10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7fa55d67a700 1 Processor -- start 2023-11-24T06:18:55.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7fa55d67a700 1 -- start start 2023-11-24T06:18:55.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7fbbe4bde700 1 Processor -- start 2023-11-24T06:18:55.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7fbbe4bde700 1 -- start start 2023-11-24T06:18:55.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f082c003a00 con 0x7f08380b4260 2023-11-24T06:18:55.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7f081ffff700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f082406af20 0x7f082406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.846+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f082c0892e0 con 0x7f08380b4260 2023-11-24T06:18:55.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7f081ffff700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f0824073140 0x7f0824075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 --> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0824075c10 con 0x7f0824073140 2023-11-24T06:18:55.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f082c089780 con 0x7f08380b4260 2023-11-24T06:18:55.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa55d67a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5580d1aa0 0x7fa5580d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa55d67a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5580d1350 0x7fa558147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa55d67a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5580d17a0 con 0x7fa558159380 2023-11-24T06:18:55.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa55d67a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5580d4ab0 con 0x7fa5580d1350 2023-11-24T06:18:55.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa55d67a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa558159760 con 0x7fa5580d1aa0 2023-11-24T06:18:55.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa556ffd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa558159380 0x7fa5580ce5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38696/0 (socket says 172.21.15.89:38696) 2023-11-24T06:18:55.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa556ffd700 1 -- 172.21.15.89:0/2765709448 learned_addr learned my addr 172.21.15.89:0/2765709448 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:55.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20d1f9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff20806d4d0 0x7ff208148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20d1f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2080b3b50 0x7ff2081617d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20d1f9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff208164a90 0x7ff208204160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20d1f9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080cf2b0 con 0x7ff2080b3b50 2023-11-24T06:18:55.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20d1f9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080cc440 con 0x7ff20806d4d0 2023-11-24T06:18:55.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20d1f9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff208148c40 con 0x7ff208164a90 2023-11-24T06:18:55.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20759e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff208164a90 0x7ff208204160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20759e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff208164a90 0x7ff208204160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44074/0 (socket says 172.21.15.89:44074) 2023-11-24T06:18:55.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff20759e700 1 -- 172.21.15.89:0/3051421833 learned_addr learned my addr 172.21.15.89:0/3051421833 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7ff206d9d700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff20806d4d0 0x7ff208148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 <== osd.5 v2:172.21.15.123:6808/410207526 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9ae8075c90 con 0x7f9ae80731c0 2023-11-24T06:18:55.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa5567fc700 1 --2- 172.21.15.89:0/2765709448 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5580d1aa0 0x7fa5580d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fa555ffb700 1 -- 172.21.15.89:0/2765709448 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163910337 0 0) 0x7fa550001490 con 0x7fa558159380 2023-11-24T06:18:55.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fa555ffb700 1 -- 172.21.15.89:0/2765709448 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5400035b0 con 0x7fa558159380 2023-11-24T06:18:55.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7ff20659c700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2080b3b50 0x7ff2081617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7ff206d9d700 1 -- 172.21.15.89:0/3051421833 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff208164a90 msgr2=0x7ff208204160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7ff206d9d700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff208164a90 0x7ff208204160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7ff206d9d700 1 -- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2080b3b50 msgr2=0x7ff2081617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7ff206d9d700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2080b3b50 0x7ff2081617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7ff206d9d700 1 -- 172.21.15.89:0/3051421833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff200009b30 con 0x7ff20806d4d0 2023-11-24T06:18:55.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7f08377fe700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f0824073140 0x7f0824075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7f08377fe700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f0824073140 0x7f0824075560 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7f0836ffd700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f082406af20 0x7f082406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7ff206d9d700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff20806d4d0 0x7ff208148700 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00e7e0 tx=0x7ff1fc00eaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fa5577fe700 1 --2- 172.21.15.89:0/2765709448 >> v2:172.21.15.89:3300/0 conn(0x7fa5580d1350 0x7fa558147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc00c070 con 0x7ff20806d4d0 2023-11-24T06:18:55.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc013610 con 0x7ff20806d4d0 2023-11-24T06:18:55.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc013980 con 0x7ff20806d4d0 2023-11-24T06:18:55.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff208148ee0 con 0x7ff20806d4d0 2023-11-24T06:18:55.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7f0836ffd700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f082406af20 0x7f082406d3e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0828006fd0 tx=0x7f0828006ba0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2081494a0 con 0x7ff20806d4d0 2023-11-24T06:18:55.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fbbe4bde700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 0x7fbbe00cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fbbe4bde700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbbe006e550 0x7fbbe0071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.847+0000 7fbbe4bde700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe006cac0 con 0x7fbbe00b3b50 2023-11-24T06:18:55.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbe4bde700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe00cd3d0 con 0x7fbbe006e550 2023-11-24T06:18:55.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbe4bde700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe006cda0 con 0x7fbbe00cc210 2023-11-24T06:18:55.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbde59c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbbe00b3b50 0x7fbbe006de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38708/0 (socket says 172.21.15.89:38708) 2023-11-24T06:18:55.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbde59c700 1 -- 172.21.15.89:0/3761229736 learned_addr learned my addr 172.21.15.89:0/3761229736 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbddd9b700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 0x7fbbe00cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbded9d700 1 --2- 172.21.15.89:0/3761229736 >> v2:172.21.15.89:3300/0 conn(0x7fbbe006e550 0x7fbbe0071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311328188 0 0) 0x7fbbe006cac0 con 0x7fbbe00b3b50 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbc80035b0 con 0x7fbbe00b3b50 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835515310 0 0) 0x7fbbc80035b0 con 0x7fbbe00b3b50 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 msgr2=0x7fbbe00cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 0x7fbbe00cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 >> v2:172.21.15.89:3300/0 conn(0x7fbbe006e550 msgr2=0x7fbbe0071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 --2- 172.21.15.89:0/3761229736 >> v2:172.21.15.89:3300/0 conn(0x7fbbe006e550 0x7fbbe0071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe006d080 con 0x7fbbe00b3b50 2023-11-24T06:18:55.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fbbd0001670 con 0x7fbbe00b3b50 2023-11-24T06:18:55.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.848+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 >> v1:172.21.15.89:6789/0 conn(0x7fbbe00b3b50 legacy=0x7fbbe006de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbdd59a700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc8003a00 0x7fbbc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbdd59a700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 0x7fbbc8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbdd59a700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc8004d90 0x7fbbc8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00cd3d0 con 0x7fbbc8004d90 2023-11-24T06:18:55.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe006cda0 con 0x7fbbc8003a00 2023-11-24T06:18:55.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc802da20 con 0x7fbbe00cc210 2023-11-24T06:18:55.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbded9d700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc8004d90 0x7fbbc8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbddd9b700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 0x7fbbc8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbddd9b700 1 -- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc8003a00 msgr2=0x7fbbc8003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbddd9b700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc8003a00 0x7fbbc8003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbddd9b700 1 -- 172.21.15.89:0/3761229736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc8004d90 msgr2=0x7fbbc8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbddd9b700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc8004d90 0x7fbbc8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.849+0000 7fbbddd9b700 1 -- 172.21.15.89:0/3761229736 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc80035b0 con 0x7fbbe00cc210 2023-11-24T06:18:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.850+0000 7fbbddd9b700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 0x7fbbc8004850 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4009b60 tx=0x7fbbc8005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dbc088310b789bb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd400e6c0 con 0x7fbbe00cc210 2023-11-24T06:18:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff208165bf0 con 0x7ff20806d4d0 2023-11-24T06:18:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fa5577fe700 1 -- 172.21.15.89:0/2765709448 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5580d1aa0 msgr2=0x7fa5580d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:55.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fa5577fe700 1 --2- 172.21.15.89:0/2765709448 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5580d1aa0 0x7fa5580d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fa5577fe700 1 -- 172.21.15.89:0/2765709448 >> v1:172.21.15.89:6789/0 conn(0x7fa558159380 legacy=0x7fa5580ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fa5577fe700 1 -- 172.21.15.89:0/2765709448 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5580d3ea0 con 0x7fa5580d1350 2023-11-24T06:18:55.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fa5577fe700 1 --2- 172.21.15.89:0/2765709448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580d1350 0x7fa558147ac0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009840 tx=0x7fa54c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10a747e66a4d8cc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd400e820 con 0x7fbbe00cc210 2023-11-24T06:18:55.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fbbdd59a700 1 -- 172.21.15.89:0/3761229736 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd400eb90 con 0x7fbbe00cc210 2023-11-24T06:18:55.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 msgr2=0x7fbbc8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fbbe4bde700 1 --2- 172.21.15.89:0/3761229736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00cc210 0x7fbbc8004850 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4009b60 tx=0x7fbbc8005880 comp rx=0 tx=0).stop 2023-11-24T06:18:55.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fbbded9d700 1 -- 172.21.15.89:0/3761229736 reap_dead start 2023-11-24T06:18:55.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3761229736 shutdown_connections 2023-11-24T06:18:55.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.851+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3761229736 >> 172.21.15.89:0/3761229736 conn(0x7fbbe00cfa30 msgr2=0x7fbbe00cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff1fc013ae0 con 0x7ff20806d4d0 2023-11-24T06:18:55.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7f68ba59c700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f68bc06d4d0 0x7f68bc14f3d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68b000c4d0 tx=0x7f68b000c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3761229736 shutdown_connections 2023-11-24T06:18:55.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3761229736 wait complete. 2023-11-24T06:18:55.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0011790 con 0x7f68bc06d4d0 2023-11-24T06:18:55.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7ff1ef7fe700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff1f406af20 0x7ff1f406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1fc0891d0 con 0x7ff20806d4d0 2023-11-24T06:18:55.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7ff1ef7fe700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7ff1f4073140 0x7ff1f4075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 --> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1f4075c10 con 0x7ff1f4073140 2023-11-24T06:18:55.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1fc08b8e0 con 0x7ff20806d4d0 2023-11-24T06:18:55.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b00118f0 con 0x7f68bc06d4d0 2023-11-24T06:18:55.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0011ca0 con 0x7f68bc06d4d0 2023-11-24T06:18:55.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68bc152210 con 0x7f68bc06d4d0 2023-11-24T06:18:55.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.852+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68bc153bf0 con 0x7f68bc06d4d0 2023-11-24T06:18:55.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.853+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68a0000fb0 con 0x7f68bc06d4d0 2023-11-24T06:18:55.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.853+0000 7fbbe4bde700 1 Processor -- start 2023-11-24T06:18:55.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.853+0000 7fbbe4bde700 1 -- start start 2023-11-24T06:18:55.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.854+0000 7ff20759e700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7ff1f4073140 0x7ff1f4075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.854+0000 7ff20659c700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff1f406af20 0x7ff1f406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.854+0000 7ff20659c700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff1f406af20 0x7ff1f406d3e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff208072400 tx=0x7ff1f8007ec0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.854+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f68b001c460 con 0x7f68bc06d4d0 2023-11-24T06:18:55.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.854+0000 7f68aaffd700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f689c06aed0 0x7f689c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68b0089330 con 0x7f68bc06d4d0 2023-11-24T06:18:55.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7f68b9d9b700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f689c06aed0 0x7f689c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7f68aaffd700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f689c0730f0 0x7f689c075510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 --> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f689c075bc0 con 0x7f689c0730f0 2023-11-24T06:18:55.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68b008b950 con 0x7f68bc06d4d0 2023-11-24T06:18:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbe4bde700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbe00cc210 0x7fbbe00aa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbe4bde700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbe00aa550 0x7fbbe00ac3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbe4bde700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00ac920 0x7fbbe00acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbe4bde700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe0157590 con 0x7fbbe00aa550 2023-11-24T06:18:55.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbe4bde700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00ab850 con 0x7fbbe00cc210 2023-11-24T06:18:55.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbe4bde700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00aacd0 con 0x7fbbe00ac920 2023-11-24T06:18:55.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbde59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbe00cc210 0x7fbbe00aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbddd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbe00aa550 0x7fbbe00ac3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbde59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbe00cc210 0x7fbbe00aa010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50336/0 (socket says 172.21.15.89:50336) 2023-11-24T06:18:55.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbde59c700 1 -- 172.21.15.89:0/3414828883 learned_addr learned my addr 172.21.15.89:0/3414828883 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7f68bad9d700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f689c0730f0 0x7f689c075510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7f68b9d9b700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f689c06aed0 0x7f689c06d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f68bc164f20 tx=0x7f68b4011000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbded9d700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00ac920 0x7fbbe00acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbde59c700 1 -- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00ac920 msgr2=0x7fbbe00acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbde59c700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbe00ac920 0x7fbbe00acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbde59c700 1 -- 172.21.15.89:0/3414828883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbe00aa550 msgr2=0x7fbbe00ac3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbde59c700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbe00aa550 0x7fbbe00ac3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.855+0000 7fbbde59c700 1 -- 172.21.15.89:0/3414828883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe00aafb0 con 0x7fbbe00cc210 2023-11-24T06:18:55.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.856+0000 7ff20759e700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7ff1f4073140 0x7ff1f4075560 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.856+0000 7f68bad9d700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f689c0730f0 0x7f689c075510 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.856+0000 7fbbde59c700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbe00cc210 0x7fbbe00aa010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000cad0 tx=0x7fbbd000cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.857+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0004870 con 0x7fbbe00cc210 2023-11-24T06:18:55.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.857+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 <== osd.0 v2:172.21.15.89:6800/1704280696 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f689c075bc0 con 0x7f689c0730f0 2023-11-24T06:18:55.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.857+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd00049d0 con 0x7fbbe00cc210 2023-11-24T06:18:55.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.857+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0004d80 con 0x7fbbe00cc210 2023-11-24T06:18:55.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.857+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 <== osd.2 v2:172.21.15.89:6816/341459167 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff1f4075c10 con 0x7ff1f4073140 2023-11-24T06:18:55.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.858+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe00ab290 con 0x7fbbe00cc210 2023-11-24T06:18:55.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.858+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe00aa9b0 con 0x7fbbe00cc210 2023-11-24T06:18:55.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.859+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 <== osd.3 v2:172.21.15.89:6824/1593118452 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0824075c10 con 0x7f0824073140 2023-11-24T06:18:55.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.859+0000 7fa555ffb700 1 -- 172.21.15.89:0/2765709448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c011830 con 0x7fa5580d1350 2023-11-24T06:18:55.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.859+0000 7fa555ffb700 1 -- 172.21.15.89:0/2765709448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54c011990 con 0x7fa5580d1350 2023-11-24T06:18:55.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.859+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbbe006d920 con 0x7fbbe00cc210 2023-11-24T06:18:55.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.859+0000 7fa555ffb700 1 -- 172.21.15.89:0/2765709448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c011d20 con 0x7fa5580d1350 2023-11-24T06:18:55.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fa55d67a700 1 -- 172.21.15.89:0/2765709448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580d1350 msgr2=0x7fa558147ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fa55d67a700 1 --2- 172.21.15.89:0/2765709448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580d1350 0x7fa558147ac0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009840 tx=0x7fa54c009b50 comp rx=0 tx=0).stop 2023-11-24T06:18:55.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fa55d67a700 1 -- 172.21.15.89:0/2765709448 shutdown_connections 2023-11-24T06:18:55.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fa55d67a700 1 --2- 172.21.15.89:0/2765709448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580d1350 0x7fa558147ac0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fa55d67a700 1 --2- 172.21.15.89:0/2765709448 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5580d1aa0 0x7fa5580d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fa55d67a700 1 -- 172.21.15.89:0/2765709448 >> 172.21.15.89:0/2765709448 conn(0x7fa55806b410 msgr2=0x7fa55806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbbd00070d0 con 0x7fbbe00cc210 2023-11-24T06:18:55.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fbbceffd700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbbc006af20 0x7fbbc006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.860+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbd00889c0 con 0x7fbbe00cc210 2023-11-24T06:18:55.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.861+0000 7fbbceffd700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fbbc0073140 0x7fbbc0075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.861+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 --> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbc0075c10 con 0x7fbbc0073140 2023-11-24T06:18:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.861+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbd008d2c0 con 0x7fbbe00cc210 2023-11-24T06:18:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.861+0000 7fbbded9d700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fbbc0073140 0x7fbbc0075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.861+0000 7fbbded9d700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fbbc0073140 0x7fbbc0075560 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.862+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 <== osd.7 v2:172.21.15.123:6800/751799151 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fbbc0075c10 con 0x7fbbc0073140 2023-11-24T06:18:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.868+0000 7fbbddd9b700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbbc006af20 0x7fbbc006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.869+0000 7fbbddd9b700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbbc006af20 0x7fbbc006d3e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4000c00 tx=0x7fbbd4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.875+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 --> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6d180751e0 con 0x7f6d04073070 2023-11-24T06:18:55.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.875+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 --> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe318002d30 con 0x7fe3200734d0 2023-11-24T06:18:55.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.875+0000 7fa55d67a700 1 -- 172.21.15.89:0/2765709448 shutdown_connections 2023-11-24T06:18:55.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.875+0000 7f6d157fa700 1 -- 172.21.15.89:0/2687908031 <== osd.1 v2:172.21.15.89:6808/4291702656 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6d10007d40 con 0x7f6d04073070 2023-11-24T06:18:55.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f6d04073070 msgr2=0x7f6d04075490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1eacf700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f6d04073070 0x7f6d04075490 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6d0406aed0 msgr2=0x7f6d0406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1eacf700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6d0406aed0 0x7f6d0406d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6d181e55a0 tx=0x7f6d080090e0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d18159f60 msgr2=0x7f6d1814fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1eacf700 1 --2- 172.21.15.89:0/2687908031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d18159f60 0x7f6d1814fb10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00b450 tx=0x7f6d0c00b760 comp rx=0 tx=0).stop 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1d06c700 1 -- 172.21.15.89:0/2687908031 reap_dead start 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 shutdown_connections 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.876+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 >> 172.21.15.89:0/2687908031 conn(0x7f6d1806cec0 msgr2=0x7f6d18071040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.878+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 shutdown_connections 2023-11-24T06:18:55.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.878+0000 7f6d1eacf700 1 -- 172.21.15.89:0/2687908031 wait complete. 2023-11-24T06:18:55.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.879+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 --> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9afc0af5b0 con 0x7f9ae80731c0 2023-11-24T06:18:55.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.880+0000 7fe326ffd700 1 -- 172.21.15.89:0/1649159492 <== osd.4 v2:172.21.15.123:6816/1261947525 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe318002d30 con 0x7fe3200734d0 2023-11-24T06:18:55.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.880+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7fe3200734d0 msgr2=0x7fe3200758f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.880+0000 7fe33cd09700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7fe3200734d0 0x7fe3200758f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.880+0000 7f9ae3fff700 1 -- 172.21.15.89:0/3560293622 <== osd.5 v2:172.21.15.123:6808/410207526 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9afc0af5b0 con 0x7f9ae80731c0 2023-11-24T06:18:55.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.881+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 --> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f68a0002d30 con 0x7f689c0730f0 2023-11-24T06:18:55.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.881+0000 7f68aaffd700 1 -- 172.21.15.89:0/4114446432 <== osd.0 v2:172.21.15.89:6800/1704280696 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f68a0002d30 con 0x7f689c0730f0 2023-11-24T06:18:55.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f9ae80731c0 msgr2=0x7f9ae80755e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f9b02053700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f9ae80731c0 0x7f9ae80755e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f689c0730f0 msgr2=0x7f689c075510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f68c0ed7700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f689c0730f0 0x7f689c075510 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f689c06aed0 msgr2=0x7f689c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f68c0ed7700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f689c06aed0 0x7f689c06d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f68bc164f20 tx=0x7f68b4011000 comp rx=0 tx=0).stop 2023-11-24T06:18:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f68bc06d4d0 msgr2=0x7f68bc14f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f68c0ed7700 1 --2- 172.21.15.89:0/4114446432 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f68bc06d4d0 0x7f68bc14f3d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68b000c4d0 tx=0x7f68b000c7e0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7f68bad9d700 1 -- 172.21.15.89:0/4114446432 reap_dead start 2023-11-24T06:18:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 shutdown_connections 2023-11-24T06:18:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 >> 172.21.15.89:0/4114446432 conn(0x7f68bc0cfa30 msgr2=0x7f68bc070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 shutdown_connections 2023-11-24T06:18:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7f68c0ed7700 1 -- 172.21.15.89:0/4114446432 wait complete. 2023-11-24T06:18:55.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.885+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9ae806afa0 msgr2=0x7f9ae806d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.885+0000 7f9b02053700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9ae806afa0 0x7f9ae806d460 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9af00096d0 tx=0x7f9af0006c10 comp rx=0 tx=0).stop 2023-11-24T06:18:55.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.885+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9afc15d6a0 msgr2=0x7f9afc140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.885+0000 7f9b02053700 1 --2- 172.21.15.89:0/3560293622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9afc15d6a0 0x7f9afc140580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9aec019040 tx=0x7f9aec004750 comp rx=0 tx=0).stop 2023-11-24T06:18:55.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.885+0000 7f9afbfff700 1 -- 172.21.15.89:0/3560293622 reap_dead start 2023-11-24T06:18:55.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.888+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 --> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbbe006d640 con 0x7fbbc0073140 2023-11-24T06:18:55.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.888+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 shutdown_connections 2023-11-24T06:18:55.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.888+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 >> 172.21.15.89:0/3560293622 conn(0x7f9afc0cfa30 msgr2=0x7f9afc0aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.888+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 --> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff20806cb60 con 0x7ff1f4073140 2023-11-24T06:18:55.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.888+0000 7ff1ef7fe700 1 -- 172.21.15.89:0/3051421833 <== osd.2 v2:172.21.15.89:6816/341459167 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff200003e70 con 0x7ff1f4073140 2023-11-24T06:18:55.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbceffd700 1 -- 172.21.15.89:0/3414828883 <== osd.7 v2:172.21.15.123:6800/751799151 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbbd8014040 con 0x7fbbc0073140 2023-11-24T06:18:55.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7ff1f4073140 msgr2=0x7ff1f4075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7ff20d1f9700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7ff1f4073140 0x7ff1f4075560 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff1f406af20 msgr2=0x7ff1f406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7ff20d1f9700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff1f406af20 0x7ff1f406d3e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff208072400 tx=0x7ff1f8007ec0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff20806d4d0 msgr2=0x7ff208148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7ff20d1f9700 1 --2- 172.21.15.89:0/3051421833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff20806d4d0 0x7ff208148700 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00e7e0 tx=0x7ff1fc00eaf0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7ff20759e700 1 -- 172.21.15.89:0/3051421833 reap_dead start 2023-11-24T06:18:55.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fbbc0073140 msgr2=0x7fbbc0075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbe4bde700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fbbc0073140 0x7fbbc0075560 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 --> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f08380686e0 con 0x7f0824073140 2023-11-24T06:18:55.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbbc006af20 msgr2=0x7fbbc006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbe4bde700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbbc006af20 0x7fbbc006d3e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4000c00 tx=0x7fbbd4008030 comp rx=0 tx=0).stop 2023-11-24T06:18:55.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbe00cc210 msgr2=0x7fbbe00aa010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbe4bde700 1 --2- 172.21.15.89:0/3414828883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbe00cc210 0x7fbbe00aa010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000cad0 tx=0x7fbbd000cde0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbded9d700 1 -- 172.21.15.89:0/3414828883 reap_dead start 2023-11-24T06:18:55.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 shutdown_connections 2023-11-24T06:18:55.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 >> 172.21.15.89:0/3414828883 conn(0x7fbbe00cfa30 msgr2=0x7fbbe00b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.879+0000 7fa55d67a700 1 -- 172.21.15.89:0/2765709448 wait complete. 2023-11-24T06:18:55.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.882+0000 7fa55d67a700 1 Processor -- start 2023-11-24T06:18:55.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.882+0000 7fa55d67a700 1 -- start start 2023-11-24T06:18:55.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa55d67a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580d1350 0x7fa5580c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa55d67a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa558159380 0x7fa558084590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa55d67a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5580c9140 0x7fa5580c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa55d67a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580855e0 con 0x7fa5580d1350 2023-11-24T06:18:55.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa55d67a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580c9b30 con 0x7fa5580c9140 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa55d67a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa558085a10 con 0x7fa558159380 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa5567fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa558159380 0x7fa558084590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa5567fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa558159380 0x7fa558084590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44114/0 (socket says 172.21.15.89:44114) 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa5567fc700 1 -- 172.21.15.89:0/616481419 learned_addr learned my addr 172.21.15.89:0/616481419 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.883+0000 7fa5577fe700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5580c9140 0x7fa5580c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7fa5567fc700 1 -- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5580c9140 msgr2=0x7fa5580c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7fa5567fc700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5580c9140 0x7fa5580c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7fa5567fc700 1 -- 172.21.15.89:0/616481419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580d1350 msgr2=0x7fa5580c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7fa5567fc700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580d1350 0x7fa5580c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.884+0000 7fa5567fc700 1 -- 172.21.15.89:0/616481419 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0094a0 con 0x7fa558159380 2023-11-24T06:18:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.888+0000 7fa5567fc700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa558159380 0x7fa558084590 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa54800e8b0 tx=0x7fa54800ebc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.888+0000 7fa53f7fe700 1 -- 172.21.15.89:0/616481419 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548013680 con 0x7fa558159380 2023-11-24T06:18:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fa53f7fe700 1 -- 172.21.15.89:0/616481419 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5480137e0 con 0x7fa558159380 2023-11-24T06:18:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa558084ad0 con 0x7fa558159380 2023-11-24T06:18:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fa53f7fe700 1 -- 172.21.15.89:0/616481419 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548013b70 con 0x7fa558159380 2023-11-24T06:18:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.889+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa558085020 con 0x7fa558159380 2023-11-24T06:18:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7f081ffff700 1 -- 172.21.15.89:0/3929724132 <== osd.3 v2:172.21.15.89:6824/1593118452 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f08380686e0 con 0x7f0824073140 2023-11-24T06:18:55.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe32006b330 msgr2=0x7fe32006d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe33cd09700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe32006b330 0x7fe32006d7f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3381fb7e0 tx=0x7fe328008030 comp rx=0 tx=0).stop 2023-11-24T06:18:55.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe338073f90 msgr2=0x7fe338152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe33cd09700 1 --2- 172.21.15.89:0/1649159492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe338073f90 0x7fe338152ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00b530 tx=0x7fe32c00b8f0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe336d9d700 1 -- 172.21.15.89:0/1649159492 reap_dead start 2023-11-24T06:18:55.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 shutdown_connections 2023-11-24T06:18:55.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 >> 172.21.15.89:0/3051421833 conn(0x7ff2080cfa30 msgr2=0x7ff2080725e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 shutdown_connections 2023-11-24T06:18:55.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fbbe4bde700 1 -- 172.21.15.89:0/3414828883 wait complete. 2023-11-24T06:18:55.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 shutdown_connections 2023-11-24T06:18:55.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 >> 172.21.15.89:0/1649159492 conn(0x7fe3380cd570 msgr2=0x7fe3380b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 shutdown_connections 2023-11-24T06:18:55.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.890+0000 7fe33cd09700 1 -- 172.21.15.89:0/1649159492 wait complete. 2023-11-24T06:18:55.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 shutdown_connections 2023-11-24T06:18:55.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7ff20d1f9700 1 -- 172.21.15.89:0/3051421833 wait complete. 2023-11-24T06:18:55.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f0824073140 msgr2=0x7f0824075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7f083da47700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f0824073140 0x7f0824075560 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f082406af20 msgr2=0x7f082406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7f083da47700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f082406af20 0x7f082406d3e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0828006fd0 tx=0x7f0828006ba0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380b4260 msgr2=0x7f08380aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7f083da47700 1 --2- 172.21.15.89:0/3929724132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380b4260 0x7f08380aecb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f082c000f60 tx=0x7f082c00c7e0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7f08377fe700 1 -- 172.21.15.89:0/3929724132 reap_dead start 2023-11-24T06:18:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.892+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 shutdown_connections 2023-11-24T06:18:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.892+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 >> 172.21.15.89:0/3929724132 conn(0x7f08380cfa30 msgr2=0x7f08380714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.892+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 shutdown_connections 2023-11-24T06:18:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.892+0000 7f083da47700 1 -- 172.21.15.89:0/3929724132 wait complete. 2023-11-24T06:18:55.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7fa53f7fe700 1 -- 172.21.15.89:0/616481419 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa548013cd0 con 0x7fa558159380 2023-11-24T06:18:55.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7fa53f7fe700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa5440aef30 0x7fa5440b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7fa53f7fe700 1 -- 172.21.15.89:0/616481419 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5480893a0 con 0x7fa558159380 2023-11-24T06:18:55.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.891+0000 7fa556ffd700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa5440aef30 0x7fa5440b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.892+0000 7fa556ffd700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa5440aef30 0x7fa5440b13f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5500012e0 tx=0x7fa550001490 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.893+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 shutdown_connections 2023-11-24T06:18:55.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.893+0000 7f9b02053700 1 -- 172.21.15.89:0/3560293622 wait complete. 2023-11-24T06:18:55.950 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2023-11-24T06:18:55.951 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-24T06:18:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.901+0000 7fa55d67a700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7fa5580c3420 0x7fa5580c3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:55.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.901+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 --> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5580d4740 con 0x7fa5580c3420 2023-11-24T06:18:55.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.901+0000 7fa5577fe700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7fa5580c3420 0x7fa5580c3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:55.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.902+0000 7fa5577fe700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7fa5580c3420 0x7fa5580c3800 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:55.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.902+0000 7fa53f7fe700 1 -- 172.21.15.89:0/616481419 <== osd.6 v2:172.21.15.123:6824/668316489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa54c022040 con 0x7fa5580c3420 2023-11-24T06:18:55.953 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2023-11-24T06:18:55.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-24T06:18:55.954 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2023-11-24T06:18:55.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-24T06:18:55.955 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2023-11-24T06:18:55.955 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2023-11-24T06:18:55.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-24T06:18:55.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-24T06:18:55.957 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2023-11-24T06:18:55.957 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-24T06:18:55.958 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2023-11-24T06:18:55.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-24T06:18:55.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.925+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 --> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa5580d4460 con 0x7fa5580c3420 2023-11-24T06:18:55.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.926+0000 7fa53f7fe700 1 -- 172.21.15.89:0/616481419 <== osd.6 v2:172.21.15.123:6824/668316489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa54c006c60 con 0x7fa5580c3420 2023-11-24T06:18:55.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.926+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7fa5580c3420 msgr2=0x7fa5580c3800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.926+0000 7fa55d67a700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7fa5580c3420 0x7fa5580c3800 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.926+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa5440aef30 msgr2=0x7fa5440b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.926+0000 7fa55d67a700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa5440aef30 0x7fa5440b13f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5500012e0 tx=0x7fa550001490 comp rx=0 tx=0).stop 2023-11-24T06:18:55.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.926+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa558159380 msgr2=0x7fa558084590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:55.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.926+0000 7fa55d67a700 1 --2- 172.21.15.89:0/616481419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa558159380 0x7fa558084590 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa54800e8b0 tx=0x7fa54800ebc0 comp rx=0 tx=0).stop 2023-11-24T06:18:55.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.927+0000 7fa5577fe700 1 -- 172.21.15.89:0/616481419 reap_dead start 2023-11-24T06:18:55.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.927+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 shutdown_connections 2023-11-24T06:18:55.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.927+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 >> 172.21.15.89:0/616481419 conn(0x7fa55806b410 msgr2=0x7fa5580b4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:55.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.927+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 shutdown_connections 2023-11-24T06:18:55.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:55.927+0000 7fa55d67a700 1 -- 172.21.15.89:0/616481419 wait complete. 2023-11-24T06:18:55.962 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2023-11-24T06:18:55.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-24T06:18:56.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.193+0000 7f1440078700 1 Processor -- start 2023-11-24T06:18:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.193+0000 7f1440078700 1 -- start start 2023-11-24T06:18:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.194+0000 7f1440078700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438166660 0x7f14380b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.194+0000 7f1440078700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1438071910 con 0x7f1438162220 2023-11-24T06:18:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.194+0000 7f1440078700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14380afb40 con 0x7f14380af6d0 2023-11-24T06:18:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.194+0000 7f1440078700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1438070fe0 con 0x7f1438166660 2023-11-24T06:18:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.193+0000 7ff8b3dd5700 1 Processor -- start 2023-11-24T06:18:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.193+0000 7ff8b3dd5700 1 -- start start 2023-11-24T06:18:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7ff8b3dd5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac0b3b50 0x7ff8ac0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7ff8b3dd5700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac06ced0 con 0x7ff8ac164a90 2023-11-24T06:18:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7ff8b3dd5700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac06da70 con 0x7ff8ac06d4d0 2023-11-24T06:18:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7ff8b3dd5700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0b4470 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7f143e615700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f14380af6d0 0x7f1438071da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:42574/0 (socket says 172.21.15.89:42574) 2023-11-24T06:18:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7f143d613700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438166660 0x7f14380b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7f143e615700 1 -- 172.21.15.89:0/2678137201 learned_addr learned my addr 172.21.15.89:0/2678137201 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7ff8b2372700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff8ac164a90 0x7ff8ac16bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38758/0 (socket says 172.21.15.89:38758) 2023-11-24T06:18:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7ff8b2372700 1 -- 172.21.15.89:0/2979434693 learned_addr learned my addr 172.21.15.89:0/2979434693 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7ff8b1b71700 1 --2- 172.21.15.89:0/2979434693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac0b3b50 0x7ff8ac0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7f143d613700 1 -- 172.21.15.89:0/2678137201 >> v1:172.21.15.123:6789/0 conn(0x7f14380af6d0 legacy=0x7f1438071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.195+0000 7f143d613700 1 -- 172.21.15.89:0/2678137201 >> v1:172.21.15.89:6789/0 conn(0x7f1438162220 legacy=0x7f1438165f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.196+0000 7f143d613700 1 -- 172.21.15.89:0/2678137201 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14380712c0 con 0x7f1438166660 2023-11-24T06:18:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.196+0000 7ff8b1b71700 1 -- 172.21.15.89:0/2979434693 >> v1:172.21.15.123:6789/0 conn(0x7ff8ac06d4d0 legacy=0x7ff8ac06eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.196+0000 7ff8b1b71700 1 -- 172.21.15.89:0/2979434693 >> v1:172.21.15.89:6789/0 conn(0x7ff8ac164a90 legacy=0x7ff8ac16bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.196+0000 7ff8b1b71700 1 -- 172.21.15.89:0/2979434693 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac1634e0 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.196+0000 7ff8b1b71700 1 --2- 172.21.15.89:0/2979434693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac0b3b50 0x7ff8ac0b3f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009870 tx=0x7ff8a8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b69a2723ace755 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.196+0000 7ff8b0b6f700 1 -- 172.21.15.89:0/2979434693 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8004230 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.196+0000 7ff8b0b6f700 1 -- 172.21.15.89:0/2979434693 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a8004390 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.196+0000 7ff8b0b6f700 1 -- 172.21.15.89:0/2979434693 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80036a0 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.197+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2979434693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac0b3b50 msgr2=0x7ff8ac0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.197+0000 7ff8b3dd5700 1 --2- 172.21.15.89:0/2979434693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac0b3b50 0x7ff8ac0b3f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009870 tx=0x7ff8a8009b80 comp rx=0 tx=0).stop 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.197+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2979434693 shutdown_connections 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.197+0000 7ff8b3dd5700 1 --2- 172.21.15.89:0/2979434693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac0b3b50 0x7ff8ac0b3f30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.197+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2979434693 >> 172.21.15.89:0/2979434693 conn(0x7ff8ac0cfa30 msgr2=0x7ff8ac0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.198+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2979434693 shutdown_connections 2023-11-24T06:18:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.198+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2979434693 wait complete. 2023-11-24T06:18:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.199+0000 7ff8b3dd5700 1 Processor -- start 2023-11-24T06:18:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.200+0000 7ff8b3dd5700 1 -- start start 2023-11-24T06:18:56.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b3dd5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8ac06d4d0 0x7ff8ac138140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b3dd5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0b3b50 0x7ff8ac14aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b3dd5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac164a90 0x7ff8ac14b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b3dd5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac14c5c0 con 0x7ff8ac06d4d0 2023-11-24T06:18:56.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b3dd5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac14c000 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b3dd5700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac14c2e0 con 0x7ff8ac164a90 2023-11-24T06:18:56.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b1370700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0b3b50 0x7ff8ac14aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b2372700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac164a90 0x7ff8ac14b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b1370700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0b3b50 0x7ff8ac14aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50374/0 (socket says 172.21.15.89:50374) 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b1370700 1 -- 172.21.15.89:0/2353773173 learned_addr learned my addr 172.21.15.89:0/2353773173 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b1370700 1 -- 172.21.15.89:0/2353773173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac164a90 msgr2=0x7ff8ac14b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b1370700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac164a90 0x7ff8ac14b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b1370700 1 -- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8ac06d4d0 msgr2=0x7ff8ac138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b1370700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8ac06d4d0 0x7ff8ac138140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.201+0000 7ff8b1370700 1 -- 172.21.15.89:0/2353773173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a80094a0 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.202+0000 7ff8b1370700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0b3b50 0x7ff8ac14aa00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00b550 tx=0x7ff89c00b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.202+0000 7ff8a27fc700 1 -- 172.21.15.89:0/2353773173 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c011830 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.202+0000 7ff8a27fc700 1 -- 172.21.15.89:0/2353773173 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89c0119b0 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.202+0000 7ff8a27fc700 1 -- 172.21.15.89:0/2353773173 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c011d60 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.202+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac14a200 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.202+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac15c800 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.203+0000 7f143d613700 1 --2- 172.21.15.89:0/2678137201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438166660 0x7f14380b5190 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1434009870 tx=0x7f1434009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c33b2259b1b4f91 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.204+0000 7ff8a27fc700 1 -- 172.21.15.89:0/2353773173 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff89c01a400 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.204+0000 7ff8a27fc700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff89806b010 0x7ff89806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.204+0000 7ff8a27fc700 1 -- 172.21.15.89:0/2353773173 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff89c089510 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.204+0000 7ff8b1b71700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff89806b010 0x7ff89806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.205+0000 7f5057bde700 1 Processor -- start 2023-11-24T06:18:56.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.205+0000 7f143ce12700 1 -- 172.21.15.89:0/2678137201 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1434011830 con 0x7f1438166660 2023-11-24T06:18:56.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.205+0000 7f143ce12700 1 -- 172.21.15.89:0/2678137201 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1434011990 con 0x7f1438166660 2023-11-24T06:18:56.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.205+0000 7f143ce12700 1 -- 172.21.15.89:0/2678137201 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1434011d20 con 0x7f1438166660 2023-11-24T06:18:56.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.206+0000 7f1440078700 1 -- 172.21.15.89:0/2678137201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438166660 msgr2=0x7f14380b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.206+0000 7f1440078700 1 --2- 172.21.15.89:0/2678137201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438166660 0x7f14380b5190 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1434009870 tx=0x7f1434009b80 comp rx=0 tx=0).stop 2023-11-24T06:18:56.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.205+0000 7fc8cf29e700 1 Processor -- start 2023-11-24T06:18:56.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.205+0000 7fc8cf29e700 1 -- start start 2023-11-24T06:18:56.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8940052e0 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.205+0000 7f5057bde700 1 -- start start 2023-11-24T06:18:56.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f5057bde700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f505015af10 0x7f505015e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f5057bde700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f50500ce140 0x7f505015ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f5057bde700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f505015f460 0x7f505015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f5057bde700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505015f8c0 con 0x7f505015f460 2023-11-24T06:18:56.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f5057bde700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50501414c0 con 0x7f50500ce140 2023-11-24T06:18:56.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f5057bde700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50500ce9f0 con 0x7f505015af10 2023-11-24T06:18:56.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cf29e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8c8162240 0x7fc8c80afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cf29e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c80ae4e0 0x7fc8c806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cf29e700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c80b46e0 con 0x7fc8c806f360 2023-11-24T06:18:56.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cf29e700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c8166f50 con 0x7fc8c8162240 2023-11-24T06:18:56.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cf29e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80b49c0 con 0x7fc8c80ae4e0 2023-11-24T06:18:56.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f505617b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f505015f460 0x7f505015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f505617b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f505015f460 0x7f505015b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35274/0 (socket says 172.21.15.89:35274) 2023-11-24T06:18:56.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f505617b700 1 -- 172.21.15.89:0/85525823 learned_addr learned my addr 172.21.15.89:0/85525823 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cd03a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8c8162240 0x7fc8c80afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cd03a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8c8162240 0x7fc8c80afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35268/0 (socket says 172.21.15.89:35268) 2023-11-24T06:18:56.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cd03a700 1 -- 172.21.15.89:0/169833962 learned_addr learned my addr 172.21.15.89:0/169833962 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f1440078700 1 -- 172.21.15.89:0/2678137201 shutdown_connections 2023-11-24T06:18:56.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f1440078700 1 --2- 172.21.15.89:0/2678137201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438166660 0x7f14380b5190 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f1440078700 1 -- 172.21.15.89:0/2678137201 >> 172.21.15.89:0/2678137201 conn(0x7f143806cec0 msgr2=0x7f143806d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cc839700 1 --2- 172.21.15.89:0/169833962 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c80ae4e0 0x7fc8c806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f5055179700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f50500ce140 0x7f505015ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7ff8b1b71700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff89806b010 0x7ff89806d4d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009bc0 tx=0x7ff8a8010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f1440078700 1 -- 172.21.15.89:0/2678137201 shutdown_connections 2023-11-24T06:18:56.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cd03a700 1 -- 172.21.15.89:0/169833962 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c80ae4e0 msgr2=0x7fc8c806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:56.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cd03a700 1 --2- 172.21.15.89:0/169833962 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c80ae4e0 0x7fc8c806c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cd03a700 1 -- 172.21.15.89:0/169833962 >> v1:172.21.15.123:6789/0 conn(0x7fc8c806f360 legacy=0x7fc8c8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7fc8cd03a700 1 -- 172.21.15.89:0/169833962 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c80b4ca0 con 0x7fc8c8162240 2023-11-24T06:18:56.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f505617b700 1 -- 172.21.15.89:0/85525823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f505015af10 msgr2=0x7f505015e9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:56.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f505617b700 1 --2- 172.21.15.89:0/85525823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f505015af10 0x7f505015e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f505617b700 1 -- 172.21.15.89:0/85525823 >> v2:172.21.15.123:3300/0 conn(0x7f50500ce140 msgr2=0x7f505015ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:56.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f505617b700 1 --2- 172.21.15.89:0/85525823 >> v2:172.21.15.123:3300/0 conn(0x7f50500ce140 0x7f505015ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f505617b700 1 -- 172.21.15.89:0/85525823 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50500cecd0 con 0x7f505015f460 2023-11-24T06:18:56.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.208+0000 7f505617b700 1 --2- 172.21.15.89:0/85525823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 0x7f505015b2f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f505006dad0 tx=0x7f504000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e339fb485ad896 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.208+0000 7fc8cd03a700 1 --2- 172.21.15.89:0/169833962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c8162240 0x7fc8c80afed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009e50 tx=0x7fc8c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a957f757770200b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.208+0000 7fc8bffff700 1 -- 172.21.15.89:0/169833962 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000e070 con 0x7fc8c8162240 2023-11-24T06:18:56.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.208+0000 7fc8bffff700 1 -- 172.21.15.89:0/169833962 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c000b910 con 0x7fc8c8162240 2023-11-24T06:18:56.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.208+0000 7fc8bffff700 1 -- 172.21.15.89:0/169833962 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000bcc0 con 0x7fc8c8162240 2023-11-24T06:18:56.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7fc8cf29e700 1 -- 172.21.15.89:0/169833962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c8162240 msgr2=0x7fc8c80afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/169833962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c8162240 0x7fc8c80afed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009e50 tx=0x7fc8c000b330 comp rx=0 tx=0).stop 2023-11-24T06:18:56.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7fc8cf29e700 1 -- 172.21.15.89:0/169833962 shutdown_connections 2023-11-24T06:18:56.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/169833962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c8162240 0x7fc8c80afed0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/169833962 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c80ae4e0 0x7fc8c806c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7fc8cf29e700 1 -- 172.21.15.89:0/169833962 >> 172.21.15.89:0/169833962 conn(0x7fc8c8071b90 msgr2=0x7fc8c8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7fc8cf29e700 1 -- 172.21.15.89:0/169833962 shutdown_connections 2023-11-24T06:18:56.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7fc8cf29e700 1 -- 172.21.15.89:0/169833962 wait complete. 2023-11-24T06:18:56.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.207+0000 7f1440078700 1 -- 172.21.15.89:0/2678137201 wait complete. 2023-11-24T06:18:56.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.210+0000 7f2a77f12700 1 Processor -- start 2023-11-24T06:18:56.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.210+0000 7fc8cf29e700 1 Processor -- start 2023-11-24T06:18:56.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.210+0000 7f2a77f12700 1 -- start start 2023-11-24T06:18:56.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.210+0000 7fc8cf29e700 1 -- start start 2023-11-24T06:18:56.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.210+0000 7f5054978700 1 -- 172.21.15.89:0/85525823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504000dc70 con 0x7f505015f460 2023-11-24T06:18:56.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.210+0000 7f5054978700 1 -- 172.21.15.89:0/85525823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504000ddd0 con 0x7f505015f460 2023-11-24T06:18:56.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.210+0000 7f5054978700 1 -- 172.21.15.89:0/85525823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040010690 con 0x7f505015f460 2023-11-24T06:18:56.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 -- 172.21.15.89:0/85525823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 msgr2=0x7f505015b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 --2- 172.21.15.89:0/85525823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 0x7f505015b2f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f505006dad0 tx=0x7f504000be70 comp rx=0 tx=0).stop 2023-11-24T06:18:56.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 -- 172.21.15.89:0/85525823 shutdown_connections 2023-11-24T06:18:56.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 --2- 172.21.15.89:0/85525823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 0x7f505015b2f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 --2- 172.21.15.89:0/85525823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f505015af10 0x7f505015e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 --2- 172.21.15.89:0/85525823 >> v2:172.21.15.123:3300/0 conn(0x7f50500ce140 0x7f505015ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 -- 172.21.15.89:0/85525823 >> 172.21.15.89:0/85525823 conn(0x7f50500cd570 msgr2=0x7f50500cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 -- 172.21.15.89:0/85525823 shutdown_connections 2023-11-24T06:18:56.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f5057bde700 1 -- 172.21.15.89:0/85525823 wait complete. 2023-11-24T06:18:56.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7fb070d0a700 1 Processor -- start 2023-11-24T06:18:56.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7fb070d0a700 1 -- start start 2023-11-24T06:18:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7fc8cf29e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c806f360 0x7fc8c809c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7fc8cf29e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8c80ae4e0 0x7fc8c80b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7fc8cf29e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c8162240 0x7fc8c809d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7fc8cf29e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c809c370 con 0x7fc8c806f360 2023-11-24T06:18:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7fc8cf29e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c815d190 con 0x7fc8c80ae4e0 2023-11-24T06:18:56.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7fc8cf29e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c809e160 con 0x7fc8c8162240 2023-11-24T06:18:56.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f2a77f12700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a700b5d10 0x7f2a700b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f2a77f12700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2a700b6e00 0x7f2a700b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f2a77f12700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a700d3b70 con 0x7f2a7015e170 2023-11-24T06:18:56.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f2a77f12700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a700cf400 con 0x7f2a700b5d10 2023-11-24T06:18:56.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.211+0000 7f2a77f12700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a700b6700 con 0x7f2a700b6e00 2023-11-24T06:18:56.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c806f360 0x7fc8c809c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c806f360 0x7fc8c809c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35278/0 (socket says 172.21.15.89:35278) 2023-11-24T06:18:56.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 -- 172.21.15.89:0/4291090068 learned_addr learned my addr 172.21.15.89:0/4291090068 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cc839700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8c80ae4e0 0x7fc8c80b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7f2a764af700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2a700b6e00 0x7f2a700b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7f2a764af700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2a700b6e00 0x7f2a700b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50394/0 (socket says 172.21.15.89:50394) 2023-11-24T06:18:56.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7f2a764af700 1 -- 172.21.15.89:0/3247761467 learned_addr learned my addr 172.21.15.89:0/3247761467 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 -- 172.21.15.89:0/4291090068 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c8162240 msgr2=0x7fc8c809d550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:18:56.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c8162240 0x7fc8c809d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 -- 172.21.15.89:0/4291090068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8c80ae4e0 msgr2=0x7fc8c80b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8c80ae4e0 0x7fc8c80b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 -- 172.21.15.89:0/4291090068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c0009b30 con 0x7fc8c806f360 2023-11-24T06:18:56.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7f2a764af700 1 -- 172.21.15.89:0/3247761467 >> v2:172.21.15.89:3300/0 conn(0x7f2a700b5d10 msgr2=0x7f2a700b61c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:18:56.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7f2a764af700 1 --2- 172.21.15.89:0/3247761467 >> v2:172.21.15.89:3300/0 conn(0x7f2a700b5d10 0x7f2a700b61c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7f2a764af700 1 -- 172.21.15.89:0/3247761467 >> v1:172.21.15.123:6789/0 conn(0x7f2a7015e170 legacy=0x7f2a700d3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7f2a764af700 1 -- 172.21.15.89:0/3247761467 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a700d2170 con 0x7f2a700b6e00 2023-11-24T06:18:56.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cd03a700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c806f360 0x7fc8c809c790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000b370 tx=0x7fc8c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fb070d0a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb06c071850 0x7fb06c071c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7f2a764af700 1 --2- 172.21.15.89:0/3247761467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a700b6e00 0x7f2a700b7230 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2a700b9920 tx=0x7f2a6800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb218f394401a6db server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fb070d0a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb06c06e5b0 0x7fb06c0adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fb070d0a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c0b4d90 con 0x7fb06c0b4560 2023-11-24T06:18:56.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fb070d0a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c0ae960 con 0x7fb06c06e5b0 2023-11-24T06:18:56.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fb070d0a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c072170 con 0x7fb06c071850 2023-11-24T06:18:56.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7f2a74cac700 1 -- 172.21.15.89:0/3247761467 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6800e070 con 0x7f2a700b6e00 2023-11-24T06:18:56.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06a59c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb06c071850 0x7fb06c071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7ff8a27fc700 1 -- 172.21.15.89:0/2353773173 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff89c087e60 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7f5057bde700 1 Processor -- start 2023-11-24T06:18:56.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7f5057bde700 1 -- start start 2023-11-24T06:18:56.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5057bde700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50500ce140 0x7f50501fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5057bde700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f505015af10 0x7f50501e5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5057bde700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 0x7f505015dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5057bde700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5050140340 con 0x7f505015f460 2023-11-24T06:18:56.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5057bde700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50501fc6c0 con 0x7f505015af10 2023-11-24T06:18:56.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5057bde700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505015b790 con 0x7f50500ce140 2023-11-24T06:18:56.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8bdffb700 1 -- 172.21.15.89:0/4291090068 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c00092e0 con 0x7fc8c806f360 2023-11-24T06:18:56.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.212+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c809bcb0 con 0x7fc8c806f360 2023-11-24T06:18:56.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c809ccd0 con 0x7fc8c806f360 2023-11-24T06:18:56.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fc8bdffb700 1 -- 172.21.15.89:0/4291090068 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c0004960 con 0x7fc8c806f360 2023-11-24T06:18:56.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fc8bdffb700 1 -- 172.21.15.89:0/4291090068 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c001b650 con 0x7fc8c806f360 2023-11-24T06:18:56.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5055179700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f505015af10 0x7f50501e5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a80052e0 con 0x7fc8c806f360 2023-11-24T06:18:56.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5055179700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f505015af10 0x7f50501e5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50408/0 (socket says 172.21.15.89:50408) 2023-11-24T06:18:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5055179700 1 -- 172.21.15.89:0/4111024438 learned_addr learned my addr 172.21.15.89:0/4111024438 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f505597a700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50500ce140 0x7f50501fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f505617b700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 0x7f505015dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5055179700 1 -- 172.21.15.89:0/4111024438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50500ce140 msgr2=0x7f50501fc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5055179700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50500ce140 0x7f50501fc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5055179700 1 -- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 msgr2=0x7f505015dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5055179700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 0x7f505015dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f5055179700 1 -- 172.21.15.89:0/4111024438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504000b7f0 con 0x7f505015af10 2023-11-24T06:18:56.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f505617b700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 0x7f505015dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:56.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.215+0000 7f5055179700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f505015af10 0x7f50501e5a50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f504800efc0 tx=0x7f504800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.209+0000 7f1440078700 1 Processor -- start 2023-11-24T06:18:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.216+0000 7f1440078700 1 -- start start 2023-11-24T06:18:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f1440078700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14380af6d0 0x7f14381633c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f1440078700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438162220 0x7f1438156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f1440078700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1438166660 0x7f1438200d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f1440078700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14382043b0 con 0x7f14380af6d0 2023-11-24T06:18:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f1440078700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14381645a0 con 0x7f1438166660 2023-11-24T06:18:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f1440078700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14380cebd0 con 0x7f1438162220 2023-11-24T06:18:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.216+0000 7fc8bdffb700 1 -- 172.21.15.89:0/4291090068 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc8c001b7b0 con 0x7fc8c806f360 2023-11-24T06:18:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7fc8bdffb700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8ac06aed0 0x7fc8ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7fc8bdffb700 1 -- 172.21.15.89:0/4291090068 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8c008af80 con 0x7fc8c806f360 2023-11-24T06:18:56.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7fc8cc839700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8ac06aed0 0x7fc8ac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.218+0000 7fc8cc839700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8ac06aed0 0x7fc8ac06d390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc8c809d1b0 tx=0x7fc8c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.218+0000 7f143de14700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14380af6d0 0x7f14381633c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.218+0000 7f143de14700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14380af6d0 0x7f14381633c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35302/0 (socket says 172.21.15.89:35302) 2023-11-24T06:18:56.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.218+0000 7f143de14700 1 -- 172.21.15.89:0/189099352 learned_addr learned my addr 172.21.15.89:0/189099352 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.218+0000 7f143d613700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438162220 0x7f1438156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.219+0000 7f143de14700 1 -- 172.21.15.89:0/189099352 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438162220 msgr2=0x7f1438156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.219+0000 7f143de14700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438162220 0x7f1438156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.219+0000 7f143de14700 1 -- 172.21.15.89:0/189099352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1438166660 msgr2=0x7f1438200d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-24T06:18:56.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.219+0000 7f143de14700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1438166660 0x7f1438200d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.219+0000 7f143de14700 1 -- 172.21.15.89:0/189099352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14340094a0 con 0x7f14380af6d0 2023-11-24T06:18:56.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f50467fc700 1 -- 172.21.15.89:0/4111024438 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504800cdc0 con 0x7f505015af10 2023-11-24T06:18:56.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f50467fc700 1 -- 172.21.15.89:0/4111024438 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504800f040 con 0x7f505015af10 2023-11-24T06:18:56.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f50467fc700 1 -- 172.21.15.89:0/4111024438 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50480146a0 con 0x7f505015af10 2023-11-24T06:18:56.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505015d170 con 0x7f505015af10 2023-11-24T06:18:56.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.217+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505015d730 con 0x7f505015af10 2023-11-24T06:18:56.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.218+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50380052e0 con 0x7f505015af10 2023-11-24T06:18:56.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.219+0000 7f143de14700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14380af6d0 0x7f14381633c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1430001ed0 tx=0x7f14300021e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.220+0000 7f50467fc700 1 -- 172.21.15.89:0/4111024438 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5048014800 con 0x7f505015af10 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06a59c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb06c071850 0x7fb06c071c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50400/0 (socket says 172.21.15.89:50400) 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06a59c700 1 -- 172.21.15.89:0/3338961026 learned_addr learned my addr 172.21.15.89:0/3338961026 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06ad9d700 1 --2- 172.21.15.89:0/3338961026 >> v2:172.21.15.89:3300/0 conn(0x7fb06c06e5b0 0x7fb06c0adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06a59c700 1 -- 172.21.15.89:0/3338961026 >> v2:172.21.15.89:3300/0 conn(0x7fb06c06e5b0 msgr2=0x7fb06c0adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06a59c700 1 --2- 172.21.15.89:0/3338961026 >> v2:172.21.15.89:3300/0 conn(0x7fb06c06e5b0 0x7fb06c0adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06a59c700 1 -- 172.21.15.89:0/3338961026 >> v1:172.21.15.123:6789/0 conn(0x7fb06c0b4560 legacy=0x7fb06c0ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06a59c700 1 -- 172.21.15.89:0/3338961026 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c070bf0 con 0x7fb06c071850 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06a59c700 1 --2- 172.21.15.89:0/3338961026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c071850 0x7fb06c071c30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb06c068fd0 tx=0x7fb06400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b9ff52f36b540c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7fb06959a700 1 -- 172.21.15.89:0/3338961026 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06400e070 con 0x7fb06c071850 2023-11-24T06:18:56.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7fb06959a700 1 -- 172.21.15.89:0/3338961026 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06400b910 con 0x7fb06c071850 2023-11-24T06:18:56.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7fb06959a700 1 -- 172.21.15.89:0/3338961026 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06400bcc0 con 0x7fb06c071850 2023-11-24T06:18:56.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7f2a74cac700 1 -- 172.21.15.89:0/3247761467 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6800b9c0 con 0x7f2a700b6e00 2023-11-24T06:18:56.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7f2a74cac700 1 -- 172.21.15.89:0/3247761467 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6800bd70 con 0x7f2a700b6e00 2023-11-24T06:18:56.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7f2a77f12700 1 -- 172.21.15.89:0/3247761467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a700b6e00 msgr2=0x7f2a700b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.213+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3247761467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a700b6e00 0x7f2a700b7230 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2a700b9920 tx=0x7f2a6800b330 comp rx=0 tx=0).stop 2023-11-24T06:18:56.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f2a77f12700 1 -- 172.21.15.89:0/3247761467 shutdown_connections 2023-11-24T06:18:56.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3247761467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a700b6e00 0x7f2a700b7230 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3247761467 >> v2:172.21.15.89:3300/0 conn(0x7f2a700b5d10 0x7f2a700b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.214+0000 7f2a77f12700 1 -- 172.21.15.89:0/3247761467 >> 172.21.15.89:0/3247761467 conn(0x7f2a700d0d10 msgr2=0x7f2a700d1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.221+0000 7f50467fc700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f503406afa0 0x7f503406d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.221+0000 7f50467fc700 1 -- 172.21.15.89:0/4111024438 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f504808aa00 con 0x7f505015af10 2023-11-24T06:18:56.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.221+0000 7f505597a700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f503406afa0 0x7f503406d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.221+0000 7f505597a700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f503406afa0 0x7f503406d460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f504c006fd0 tx=0x7f504c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.222+0000 7fb070d0a700 1 -- 172.21.15.89:0/3338961026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c071850 msgr2=0x7fb06c071c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.222+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3338961026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c071850 0x7fb06c071c30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb06c068fd0 tx=0x7fb06400b330 comp rx=0 tx=0).stop 2023-11-24T06:18:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.222+0000 7fb070d0a700 1 -- 172.21.15.89:0/3338961026 shutdown_connections 2023-11-24T06:18:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.222+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3338961026 >> v2:172.21.15.89:3300/0 conn(0x7fb06c06e5b0 0x7fb06c0adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.222+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3338961026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c071850 0x7fb06c071c30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.222+0000 7fb070d0a700 1 -- 172.21.15.89:0/3338961026 >> 172.21.15.89:0/3338961026 conn(0x7fb06c0cfa30 msgr2=0x7fb06c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.222+0000 7fb070d0a700 1 -- 172.21.15.89:0/3338961026 shutdown_connections 2023-11-24T06:18:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.222+0000 7fb070d0a700 1 -- 172.21.15.89:0/3338961026 wait complete. 2023-11-24T06:18:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.223+0000 7fb070d0a700 1 Processor -- start 2023-11-24T06:18:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.223+0000 7fb070d0a700 1 -- start start 2023-11-24T06:18:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7f9fb9622700 1 Processor -- start 2023-11-24T06:18:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7f9fb9622700 1 -- start start 2023-11-24T06:18:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7f50467fc700 1 -- 172.21.15.89:0/4111024438 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5048055a90 con 0x7f505015af10 2023-11-24T06:18:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb070d0a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c06e5b0 0x7fb06c166ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb070d0a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb06c071850 0x7fb06c0cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7f142e7fc700 1 -- 172.21.15.89:0/189099352 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430003c00 con 0x7f14380af6d0 2023-11-24T06:18:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7f142e7fc700 1 -- 172.21.15.89:0/189099352 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1430003d60 con 0x7f14380af6d0 2023-11-24T06:18:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7f142e7fc700 1 -- 172.21.15.89:0/189099352 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143000abd0 con 0x7f14380af6d0 2023-11-24T06:18:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb070d0a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c0b4560 0x7fb06c16c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb070d0a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c16e8e0 con 0x7fb06c06e5b0 2023-11-24T06:18:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb070d0a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0ce710 con 0x7fb06c0b4560 2023-11-24T06:18:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb070d0a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0ce9f0 con 0x7fb06c071850 2023-11-24T06:18:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb069d9b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb06c071850 0x7fb06c0cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f9fb9622700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9fb40b0fa0 0x7f9fb40afc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fc8bdffb700 1 -- 172.21.15.89:0/4291090068 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8c00561a0 con 0x7fc8c806f360 2023-11-24T06:18:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143815fdb0 con 0x7f14380af6d0 2023-11-24T06:18:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1438160370 con 0x7f14380af6d0 2023-11-24T06:18:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f2a77f12700 1 -- 172.21.15.89:0/3247761467 shutdown_connections 2023-11-24T06:18:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.226+0000 7f2a77f12700 1 -- 172.21.15.89:0/3247761467 wait complete. 2023-11-24T06:18:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.226+0000 7fc4fa0f3700 1 Processor -- start 2023-11-24T06:18:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.226+0000 7fc4fa0f3700 1 -- start start 2023-11-24T06:18:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1438067f60 con 0x7f14380af6d0 2023-11-24T06:18:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f9fb9622700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9fb40ca190 0x7f9fb40ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f9fb9622700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb40ca620 con 0x7f9fb40d0c40 2023-11-24T06:18:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f9fb9622700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb40d10a0 con 0x7f9fb40ca190 2023-11-24T06:18:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f9fb9622700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb40622a0 con 0x7f9fb40b0fa0 2023-11-24T06:18:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f9fb2ffd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9fb40b0fa0 0x7f9fb40afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f9fb2ffd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9fb40b0fa0 0x7f9fb40afc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50426/0 (socket says 172.21.15.89:50426) 2023-11-24T06:18:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7f9fb2ffd700 1 -- 172.21.15.89:0/4019099319 learned_addr learned my addr 172.21.15.89:0/4019099319 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.226+0000 7f9fb2ffd700 1 -- 172.21.15.89:0/4019099319 >> v2:172.21.15.89:3300/0 conn(0x7f9fb40ca190 msgr2=0x7f9fb40ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.226+0000 7f9fb2ffd700 1 --2- 172.21.15.89:0/4019099319 >> v2:172.21.15.89:3300/0 conn(0x7f9fb40ca190 0x7f9fb40ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.226+0000 7f9fb2ffd700 1 -- 172.21.15.89:0/4019099319 >> v1:172.21.15.89:6789/0 conn(0x7f9fb40d0c40 legacy=0x7f9fb40b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.226+0000 7f9fb2ffd700 1 -- 172.21.15.89:0/4019099319 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb4062580 con 0x7f9fb40b0fa0 2023-11-24T06:18:56.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7f2a77f12700 1 Processor -- start 2023-11-24T06:18:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fc4fa0f3700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc4f4140f20 0x7fc4f40b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fc4fa0f3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4f40b2760 0x7fc4f415e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fc4fa0f3700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f4161b40 con 0x7fc4f40b3900 2023-11-24T06:18:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fc4fa0f3700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f40b3e10 con 0x7fc4f4140f20 2023-11-24T06:18:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fc4fa0f3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f4161d90 con 0x7fc4f40b2760 2023-11-24T06:18:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fc4f37fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc4f4140f20 0x7fc4f40b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fc4f37fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc4f4140f20 0x7fc4f40b33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50432/0 (socket says 172.21.15.89:50432) 2023-11-24T06:18:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fc4f37fe700 1 -- 172.21.15.89:0/2724840616 learned_addr learned my addr 172.21.15.89:0/2724840616 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7f2a77f12700 1 -- start start 2023-11-24T06:18:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7fc4f37fe700 1 -- 172.21.15.89:0/2724840616 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4f40b2760 msgr2=0x7fc4f415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7fc4f37fe700 1 --2- 172.21.15.89:0/2724840616 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4f40b2760 0x7fc4f415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7fc4f37fe700 1 -- 172.21.15.89:0/2724840616 >> v1:172.21.15.89:6789/0 conn(0x7fc4f40b3900 legacy=0x7fc4f40b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7fc4f37fe700 1 -- 172.21.15.89:0/2724840616 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f4162070 con 0x7fc4f4140f20 2023-11-24T06:18:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7fc4f37fe700 1 --2- 172.21.15.89:0/2724840616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f4140f20 0x7fc4f40b33c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc4e401a990 tx=0x7fc4e401aca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=579a03dc228053b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7fc4f27fc700 1 -- 172.21.15.89:0/2724840616 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e401c870 con 0x7fc4f4140f20 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a77f12700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a700b6e00 0x7f2a70148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a77f12700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a7015e170 0x7f2a700c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a77f12700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a700c6610 0x7f2a701549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a77f12700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a701563d0 con 0x7f2a700c6610 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a77f12700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a700cbfc0 con 0x7f2a7015e170 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a77f12700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7014cdd0 con 0x7f2a700b6e00 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a75cae700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a700b6e00 0x7f2a70148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a75cae700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a700b6e00 0x7f2a70148ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44202/0 (socket says 172.21.15.89:44202) 2023-11-24T06:18:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a754ad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a7015e170 0x7f2a700c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a75cae700 1 -- 172.21.15.89:0/3474381385 learned_addr learned my addr 172.21.15.89:0/3474381385 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7f2a764af700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a700c6610 0x7f2a701549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7fc4f27fc700 1 -- 172.21.15.89:0/2724840616 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e401c9d0 con 0x7fc4f4140f20 2023-11-24T06:18:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7fc4f27fc700 1 -- 172.21.15.89:0/2724840616 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e401cd60 con 0x7fc4f4140f20 2023-11-24T06:18:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2724840616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f4140f20 msgr2=0x7fc4f40b33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.229+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2724840616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f4140f20 0x7fc4f40b33c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc4e401a990 tx=0x7fc4e401aca0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a754ad700 1 -- 172.21.15.89:0/3474381385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a700b6e00 msgr2=0x7f2a70148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a754ad700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a700b6e00 0x7f2a70148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a754ad700 1 -- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a700c6610 msgr2=0x7f2a701549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a754ad700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a700c6610 0x7f2a701549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a754ad700 1 -- 172.21.15.89:0/3474381385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a68009b30 con 0x7f2a7015e170 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a754ad700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a7015e170 0x7f2a700c60d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2a700b60f0 tx=0x7f2a6c00b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a667fc700 1 -- 172.21.15.89:0/3474381385 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c011620 con 0x7f2a7015e170 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a667fc700 1 -- 172.21.15.89:0/3474381385 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6c011780 con 0x7f2a7015e170 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7014d0b0 con 0x7f2a7015e170 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a667fc700 1 -- 172.21.15.89:0/3474381385 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c011b10 con 0x7f2a7015e170 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.230+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7014d440 con 0x7f2a7015e170 2023-11-24T06:18:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.231+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a700cc170 con 0x7f2a7015e170 2023-11-24T06:18:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2724840616 shutdown_connections 2023-11-24T06:18:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2724840616 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4f40b2760 0x7fc4f415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2724840616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f4140f20 0x7fc4f40b33c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2724840616 >> 172.21.15.89:0/2724840616 conn(0x7fc4f40743b0 msgr2=0x7fc4f406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7f2a667fc700 1 -- 172.21.15.89:0/3474381385 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2a6c011c70 con 0x7f2a7015e170 2023-11-24T06:18:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f2a667fc700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a5406aed0 0x7f2a5406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f2a667fc700 1 -- 172.21.15.89:0/3474381385 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a6c089360 con 0x7f2a7015e170 2023-11-24T06:18:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f2a75cae700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a5406aed0 0x7f2a5406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f2a75cae700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a5406aed0 0x7f2a5406d390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2a60001920 tx=0x7f2a60009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.234+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2724840616 shutdown_connections 2023-11-24T06:18:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.234+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2724840616 wait complete. 2023-11-24T06:18:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7f9fb2ffd700 1 --2- 172.21.15.89:0/4019099319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40b0fa0 0x7f9fb40afc80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9fb406fc90 tx=0x7f9fa40134c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de75081c30fe626a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7f9fb1ffb700 1 -- 172.21.15.89:0/4019099319 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4016070 con 0x7f9fb40b0fa0 2023-11-24T06:18:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7f9fb1ffb700 1 -- 172.21.15.89:0/4019099319 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa4013dc0 con 0x7f9fb40b0fa0 2023-11-24T06:18:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7f9fb1ffb700 1 -- 172.21.15.89:0/4019099319 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4004250 con 0x7f9fb40b0fa0 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f9fb9622700 1 -- 172.21.15.89:0/4019099319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40b0fa0 msgr2=0x7f9fb40afc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f9fb9622700 1 --2- 172.21.15.89:0/4019099319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40b0fa0 0x7f9fb40afc80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9fb406fc90 tx=0x7f9fa40134c0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f9fb9622700 1 -- 172.21.15.89:0/4019099319 shutdown_connections 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f9fb9622700 1 --2- 172.21.15.89:0/4019099319 >> v2:172.21.15.89:3300/0 conn(0x7f9fb40ca190 0x7f9fb40ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f9fb9622700 1 --2- 172.21.15.89:0/4019099319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40b0fa0 0x7f9fb40afc80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f9fb9622700 1 -- 172.21.15.89:0/4019099319 >> 172.21.15.89:0/4019099319 conn(0x7f9fb40b57b0 msgr2=0x7f9fb40b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7f9fb9622700 1 -- 172.21.15.89:0/4019099319 shutdown_connections 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb069d9b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb06c071850 0x7fb06c0cdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44186/0 (socket says 172.21.15.89:44186) 2023-11-24T06:18:56.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.224+0000 7fb069d9b700 1 -- 172.21.15.89:0/3394254140 learned_addr learned my addr 172.21.15.89:0/3394254140 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb069d9b700 1 -- 172.21.15.89:0/3394254140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c0b4560 msgr2=0x7fb06c16c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb069d9b700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c0b4560 0x7fb06c16c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb069d9b700 1 -- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c06e5b0 msgr2=0x7fb06c166ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb069d9b700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c06e5b0 0x7fb06c166ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb069d9b700 1 -- 172.21.15.89:0/3394254140 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb064009b30 con 0x7fb06c071850 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb069d9b700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb06c071850 0x7fb06c0cdda0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00c420 tx=0x7fb05c00c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb05affd700 1 -- 172.21.15.89:0/3394254140 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0052d0 con 0x7fb06c071850 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb05affd700 1 -- 172.21.15.89:0/3394254140 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c005430 con 0x7fb06c071850 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb05affd700 1 -- 172.21.15.89:0/3394254140 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0038a0 con 0x7fb06c071850 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.225+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c0cecd0 con 0x7fb06c071850 2023-11-24T06:18:56.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.226+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c0ce2e0 con 0x7fb06c071850 2023-11-24T06:18:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fb05affd700 1 -- 172.21.15.89:0/3394254140 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb05c003a00 con 0x7fb06c071850 2023-11-24T06:18:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.227+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c06f510 con 0x7fb06c071850 2023-11-24T06:18:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7fb05affd700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb04c06aed0 0x7fb04c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.228+0000 7fb05affd700 1 -- 172.21.15.89:0/3394254140 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb05c088960 con 0x7fb06c071850 2023-11-24T06:18:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.232+0000 7fb06a59c700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb04c06aed0 0x7fb04c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7fb06a59c700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb04c06aed0 0x7fb04c06d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb06c08b960 tx=0x7fb064010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.233+0000 7fb05affd700 1 -- 172.21.15.89:0/3394254140 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb05c053aa0 con 0x7fb06c071850 2023-11-24T06:18:56.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.235+0000 7fc4fa0f3700 1 Processor -- start 2023-11-24T06:18:56.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.236+0000 7f142e7fc700 1 -- 172.21.15.89:0/189099352 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f143000ad30 con 0x7f14380af6d0 2023-11-24T06:18:56.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7f9fb9622700 1 -- 172.21.15.89:0/4019099319 wait complete. 2023-11-24T06:18:56.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7f142e7fc700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f141c06af20 0x7f141c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7f142e7fc700 1 -- 172.21.15.89:0/189099352 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f143008a200 con 0x7f14380af6d0 2023-11-24T06:18:56.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7f142e7fc700 1 -- 172.21.15.89:0/189099352 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f143008a5c0 con 0x7f14380af6d0 2023-11-24T06:18:56.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7f2a667fc700 1 -- 172.21.15.89:0/3474381385 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a700cc170 con 0x7f2a7015e170 2023-11-24T06:18:56.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.236+0000 7fc4fa0f3700 1 -- start start 2023-11-24T06:18:56.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4fa0f3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f40b2760 0x7fc4f41403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4fa0f3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f40b3900 0x7fc4f4161260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4fa0f3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4f4140f20 0x7fc4f415e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4fa0f3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40cde00 con 0x7fc4f40b2760 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4fa0f3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40cd180 con 0x7fc4f40b3900 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4fa0f3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40cd460 con 0x7fc4f4140f20 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4f37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f40b2760 0x7fc4f41403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4f37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f40b2760 0x7fc4f41403d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35336/0 (socket says 172.21.15.89:35336) 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4f37fe700 1 -- 172.21.15.89:0/2681590714 learned_addr learned my addr 172.21.15.89:0/2681590714 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.237+0000 7fc4f2ffd700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f40b3900 0x7fc4f4161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7fc4f2ffd700 1 -- 172.21.15.89:0/2681590714 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4f4140f20 msgr2=0x7fc4f415e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7fc4f2ffd700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4f4140f20 0x7fc4f415e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7fc4f2ffd700 1 -- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f40b2760 msgr2=0x7fc4f41403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7fc4f2ffd700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f40b2760 0x7fc4f41403d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7fc4f2ffd700 1 -- 172.21.15.89:0/2681590714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e401a5f0 con 0x7fc4f40b3900 2023-11-24T06:18:56.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7fc4f2ffd700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f40b3900 0x7fc4f4161260 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8000f60 tx=0x7fc4e800c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7f9fb9622700 1 Processor -- start 2023-11-24T06:18:56.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.238+0000 7f9fb9622700 1 -- start start 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb9622700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb40b0fa0 0x7f9fb414f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb9622700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fb40ca190 0x7f9fb41f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb9622700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40d0c40 0x7f9fb41f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb9622700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb414fdb0 con 0x7f9fb40ca190 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb9622700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb4154af0 con 0x7f9fb40d0c40 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb9622700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb414f8b0 con 0x7f9fb40b0fa0 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb27fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fb40ca190 0x7f9fb41f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb27fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fb40ca190 0x7f9fb41f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35344/0 (socket says 172.21.15.89:35344) 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.239+0000 7f9fb27fc700 1 -- 172.21.15.89:0/3339713072 learned_addr learned my addr 172.21.15.89:0/3339713072 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb37fe700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40d0c40 0x7f9fb41f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb2ffd700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb40b0fa0 0x7f9fb414f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb27fc700 1 -- 172.21.15.89:0/3339713072 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb40b0fa0 msgr2=0x7f9fb414f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb27fc700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb40b0fa0 0x7f9fb414f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb27fc700 1 -- 172.21.15.89:0/3339713072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40d0c40 msgr2=0x7f9fb41f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb27fc700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40d0c40 0x7f9fb41f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb27fc700 1 -- 172.21.15.89:0/3339713072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa4011df0 con 0x7f9fb40ca190 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb27fc700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fb40ca190 0x7f9fb41f3250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800b4a0 tx=0x7f9fa800b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9f9b7fe700 1 -- 172.21.15.89:0/3339713072 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8011830 con 0x7f9fb40ca190 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb4155950 con 0x7f9fb40ca190 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb4155020 con 0x7f9fb40ca190 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9f9b7fe700 1 -- 172.21.15.89:0/3339713072 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa800be70 con 0x7f9fb40ca190 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.240+0000 7f9f9b7fe700 1 -- 172.21.15.89:0/3339713072 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8011b90 con 0x7f9fb40ca190 2023-11-24T06:18:56.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.243+0000 7f9f9b7fe700 1 -- 172.21.15.89:0/3339713072 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9fa8011cf0 con 0x7f9fb40ca190 2023-11-24T06:18:56.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.243+0000 7f9f9b7fe700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9fa006aed0 0x7f9fa006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.243+0000 7f9f9b7fe700 1 -- 172.21.15.89:0/3339713072 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9fa80893e0 con 0x7f9fb40ca190 2023-11-24T06:18:56.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.243+0000 7f9fb2ffd700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9fa006aed0 0x7f9fa006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.244+0000 7f9fb2ffd700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9fa006aed0 0x7f9fa006d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4074470 tx=0x7f9fa4011000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.247+0000 7f143d613700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f141c06af20 0x7f141c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.249+0000 7fc4dbfff700 1 -- 172.21.15.89:0/2681590714 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e80052d0 con 0x7fc4f40b3900 2023-11-24T06:18:56.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.249+0000 7fc4dbfff700 1 -- 172.21.15.89:0/2681590714 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e8005430 con 0x7fc4f40b3900 2023-11-24T06:18:56.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.249+0000 7fc4dbfff700 1 -- 172.21.15.89:0/2681590714 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e80038c0 con 0x7fc4f40b3900 2023-11-24T06:18:56.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.249+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f40cd740 con 0x7fc4f40b3900 2023-11-24T06:18:56.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.249+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f40cfc30 con 0x7fc4f40b3900 2023-11-24T06:18:56.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.250+0000 7f143d613700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f141c06af20 0x7f141c06d3e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f14340090b0 tx=0x7f143401d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.251+0000 7fc4dbfff700 1 -- 172.21.15.89:0/2681590714 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc4e8003a20 con 0x7fc4f40b3900 2023-11-24T06:18:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.251+0000 7fc4dbfff700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4dc06b060 0x7fc4dc06d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.252+0000 7fc4dbfff700 1 -- 172.21.15.89:0/2681590714 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4e80889a0 con 0x7fc4f40b3900 2023-11-24T06:18:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.253+0000 7fc4f37fe700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4dc06b060 0x7fc4dc06d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.253+0000 7fc4f37fe700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4dc06b060 0x7fc4dc06d520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc4e401d010 tx=0x7fc4e4022000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.260+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f940052e0 con 0x7f9fb40ca190 2023-11-24T06:18:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.266+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d40052e0 con 0x7fc4f40b3900 2023-11-24T06:18:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.270+0000 7f9f9b7fe700 1 -- 172.21.15.89:0/3339713072 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fa8054490 con 0x7f9fb40ca190 2023-11-24T06:18:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.272+0000 7fc4dbfff700 1 -- 172.21.15.89:0/2681590714 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4e8053b30 con 0x7fc4f40b3900 2023-11-24T06:18:56.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.613+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff894005860 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.613+0000 7ff8a27fc700 1 -- 172.21.15.89:0/2353773173 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff89c015030 con 0x7ff8ac0b3b50 2023-11-24T06:18:56.618 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2023-11-24T06:18:56.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff89806b010 msgr2=0x7ff89806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff89806b010 0x7ff89806d4d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009bc0 tx=0x7ff8a8010000 comp rx=0 tx=0).stop 2023-11-24T06:18:56.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0b3b50 msgr2=0x7ff8ac14aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0b3b50 0x7ff8ac14aa00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00b550 tx=0x7ff89c00b860 comp rx=0 tx=0).stop 2023-11-24T06:18:56.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 shutdown_connections 2023-11-24T06:18:56.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac164a90 0x7ff8ac14b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8ac06d4d0 0x7ff8ac138140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff89806b010 0x7ff89806d4d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 --2- 172.21.15.89:0/2353773173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0b3b50 0x7ff8ac14aa00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.617+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 >> 172.21.15.89:0/2353773173 conn(0x7ff8ac0cfa30 msgr2=0x7ff8ac16e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.618+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 shutdown_connections 2023-11-24T06:18:56.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.618+0000 7ff8b3dd5700 1 -- 172.21.15.89:0/2353773173 wait complete. 2023-11-24T06:18:56.628 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.622+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc8a8005860 con 0x7fc8c806f360 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.622+0000 7fc8bdffb700 1 -- 172.21.15.89:0/4291090068 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc8c00597c0 con 0x7fc8c806f360 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.626+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8ac06aed0 msgr2=0x7fc8ac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.626+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8ac06aed0 0x7fc8ac06d390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc8c809d1b0 tx=0x7fc8c4008040 comp rx=0 tx=0).stop 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.626+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c806f360 msgr2=0x7fc8c809c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.626+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c806f360 0x7fc8c809c790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000b370 tx=0x7fc8c0004ad0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.627+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 shutdown_connections 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.627+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c8162240 0x7fc8c809d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.627+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c806f360 0x7fc8c809c790 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.627+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8ac06aed0 0x7fc8ac06d390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.627+0000 7fc8cf29e700 1 --2- 172.21.15.89:0/4291090068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8c80ae4e0 0x7fc8c80b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.627+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 >> 172.21.15.89:0/4291090068 conn(0x7fc8c8071b90 msgr2=0x7fc8c806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.628+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 shutdown_connections 2023-11-24T06:18:56.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.628+0000 7fc8cf29e700 1 -- 172.21.15.89:0/4291090068 wait complete. 2023-11-24T06:18:56.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.629+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f5038005860 con 0x7f505015af10 2023-11-24T06:18:56.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.633+0000 7f50467fc700 1 -- 172.21.15.89:0/4111024438 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5048018090 con 0x7f505015af10 2023-11-24T06:18:56.636 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2023-11-24T06:18:56.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.633+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb06c0ceb30 con 0x7fb06c071850 2023-11-24T06:18:56.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.635+0000 7fb05affd700 1 -- 172.21.15.89:0/3394254140 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb06c0ceb30 con 0x7fb06c071850 2023-11-24T06:18:56.642 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2023-11-24T06:18:56.642 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2023-11-24T06:18:56.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb04c06aed0 msgr2=0x7fb04c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb04c06aed0 0x7fb04c06d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb06c08b960 tx=0x7fb064010000 comp rx=0 tx=0).stop 2023-11-24T06:18:56.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb06c071850 msgr2=0x7fb06c0cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb06c071850 0x7fb06c0cdda0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00c420 tx=0x7fb05c00c7e0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f503406afa0 msgr2=0x7f503406d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7f5057bde700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f503406afa0 0x7f503406d460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f504c006fd0 tx=0x7f504c009190 comp rx=0 tx=0).stop 2023-11-24T06:18:56.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f505015af10 msgr2=0x7f50501e5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7f5057bde700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f505015af10 0x7f50501e5a50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f504800efc0 tx=0x7f504800c3a0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 shutdown_connections 2023-11-24T06:18:56.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb04c06aed0 0x7fb04c06d390 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb06c08b960 tx=0x7fb064010000 comp rx=0 tx=0).stop 2023-11-24T06:18:56.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb06c0b4560 0x7fb06c16c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c06e5b0 0x7fb06c166ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 --2- 172.21.15.89:0/3394254140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb06c071850 0x7fb06c0cdda0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 >> 172.21.15.89:0/3394254140 conn(0x7fb06c0cfa30 msgr2=0x7fb06c0cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 shutdown_connections 2023-11-24T06:18:56.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7fb070d0a700 1 -- 172.21.15.89:0/3394254140 wait complete. 2023-11-24T06:18:56.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.639+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2a70067f60 con 0x7f2a7015e170 2023-11-24T06:18:56.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 shutdown_connections 2023-11-24T06:18:56.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f5057bde700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f505015f460 0x7f505015dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f5057bde700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50500ce140 0x7f50501fc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f5057bde700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f503406afa0 0x7f503406d460 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f5057bde700 1 --2- 172.21.15.89:0/4111024438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f505015af10 0x7f50501e5a50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 >> 172.21.15.89:0/4111024438 conn(0x7f50500cd570 msgr2=0x7f50500b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 shutdown_connections 2023-11-24T06:18:56.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f2a667fc700 1 -- 172.21.15.89:0/3474381385 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2a6c054410 con 0x7f2a7015e170 2023-11-24T06:18:56.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.640+0000 7f5057bde700 1 -- 172.21.15.89:0/4111024438 wait complete. 2023-11-24T06:18:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2023-11-24T06:18:56.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.643+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f14380b5860 con 0x7f14380af6d0 2023-11-24T06:18:56.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a5406aed0 msgr2=0x7f2a5406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a5406aed0 0x7f2a5406d390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2a60001920 tx=0x7f2a60009040 comp rx=0 tx=0).stop 2023-11-24T06:18:56.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a7015e170 msgr2=0x7f2a700c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a7015e170 0x7f2a700c60d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2a700b60f0 tx=0x7f2a6c00b730 comp rx=0 tx=0).stop 2023-11-24T06:18:56.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 shutdown_connections 2023-11-24T06:18:56.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a700c6610 0x7f2a701549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a700b6e00 0x7f2a70148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a5406aed0 0x7f2a5406d390 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 --2- 172.21.15.89:0/3474381385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a7015e170 0x7f2a700c60d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 >> 172.21.15.89:0/3474381385 conn(0x7f2a700d0d10 msgr2=0x7f2a700c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 shutdown_connections 2023-11-24T06:18:56.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.644+0000 7f2a77f12700 1 -- 172.21.15.89:0/3474381385 wait complete. 2023-11-24T06:18:56.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.645+0000 7f142e7fc700 1 -- 172.21.15.89:0/189099352 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1430055290 con 0x7f14380af6d0 2023-11-24T06:18:56.650 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2023-11-24T06:18:56.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f141c06af20 msgr2=0x7f141c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f141c06af20 0x7f141c06d3e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f14340090b0 tx=0x7f143401d040 comp rx=0 tx=0).stop 2023-11-24T06:18:56.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14380af6d0 msgr2=0x7f14381633c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14380af6d0 0x7f14381633c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1430001ed0 tx=0x7f14300021e0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 shutdown_connections 2023-11-24T06:18:56.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f141c06af20 0x7f141c06d3e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1438166660 0x7f1438200d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14380af6d0 0x7f14381633c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 --2- 172.21.15.89:0/189099352 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1438162220 0x7f1438156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.649+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 >> 172.21.15.89:0/189099352 conn(0x7f143806cec0 msgr2=0x7f143815cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2023-11-24T06:18:56.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.651+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 shutdown_connections 2023-11-24T06:18:56.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.651+0000 7f1440078700 1 -- 172.21.15.89:0/189099352 wait complete. 2023-11-24T06:18:56.664 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-11-24T06:18:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.661+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9f94005860 con 0x7f9fb40ca190 2023-11-24T06:18:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.662+0000 7f9f9b7fe700 1 -- 172.21.15.89:0/3339713072 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9fa8015070 con 0x7f9fb40ca190 2023-11-24T06:18:56.664 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2023-11-24T06:18:56.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2023-11-24T06:18:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9fa006aed0 msgr2=0x7f9fa006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9fa006aed0 0x7f9fa006d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4074470 tx=0x7f9fa4011000 comp rx=0 tx=0).stop 2023-11-24T06:18:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fb40ca190 msgr2=0x7f9fb41f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fb40ca190 0x7f9fb41f3250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800b4a0 tx=0x7f9fa800b7b0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 shutdown_connections 2023-11-24T06:18:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9fa006aed0 0x7f9fa006d390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fb40d0c40 0x7f9fb41f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb40b0fa0 0x7f9fb414f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 --2- 172.21.15.89:0/3339713072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fb40ca190 0x7f9fb41f3250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.664+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 >> 172.21.15.89:0/3339713072 conn(0x7f9fb40b57b0 msgr2=0x7f9fb415b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.665+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 shutdown_connections 2023-11-24T06:18:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.665+0000 7f9fb9622700 1 -- 172.21.15.89:0/3339713072 wait complete. 2023-11-24T06:18:56.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2023-11-24T06:18:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.666+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc4d4005860 con 0x7fc4f40b3900 2023-11-24T06:18:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.667+0000 7fc4dbfff700 1 -- 172.21.15.89:0/2681590714 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc4e8017030 con 0x7fc4f40b3900 2023-11-24T06:18:56.669 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2023-11-24T06:18:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4dc06b060 msgr2=0x7fc4dc06d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4dc06b060 0x7fc4dc06d520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc4e401d010 tx=0x7fc4e4022000 comp rx=0 tx=0).stop 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f40b3900 msgr2=0x7fc4f4161260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f40b3900 0x7fc4f4161260 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8000f60 tx=0x7fc4e800c7e0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 shutdown_connections 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4f4140f20 0x7fc4f415e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f40b2760 0x7fc4f41403d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4dc06b060 0x7fc4dc06d520 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 --2- 172.21.15.89:0/2681590714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4f40b3900 0x7fc4f4161260 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 >> 172.21.15.89:0/2681590714 conn(0x7fc4f40743b0 msgr2=0x7fc4f4143c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 shutdown_connections 2023-11-24T06:18:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:56.669+0000 7fc4fa0f3700 1 -- 172.21.15.89:0/2681590714 wait complete. 2023-11-24T06:18:56.673 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2023-11-24T06:18:56.681 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2023-11-24T06:18:56.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2023-11-24T06:18:57.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-24T06:18:57.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-24T06:18:57.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-24T06:18:57.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-24T06:18:57.670 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-24T06:18:57.674 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-24T06:18:57.683 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-24T06:18:57.685 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-24T06:18:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.827+0000 7f24817f2700 1 Processor -- start 2023-11-24T06:18:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.827+0000 7f24817f2700 1 -- start start 2023-11-24T06:18:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.828+0000 7f24817f2700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f247c15d6a0 0x7f247c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.828+0000 7f24817f2700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c0720e0 con 0x7f247c0b4090 2023-11-24T06:18:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.828+0000 7f24817f2700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c0723c0 con 0x7f247c071be0 2023-11-24T06:18:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.828+0000 7f24817f2700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c072d60 con 0x7f247c15d6a0 2023-11-24T06:18:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.829+0000 7f247affd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f247c15d6a0 0x7f247c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.829+0000 7f247a7fc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f247c0b4090 0x7f247c0714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38860/0 (socket says 172.21.15.89:38860) 2023-11-24T06:18:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.829+0000 7f247a7fc700 1 -- 172.21.15.89:0/1248820258 learned_addr learned my addr 172.21.15.89:0/1248820258 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.830+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645393357 0 0) 0x7f247c0720e0 con 0x7f247c0b4090 2023-11-24T06:18:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.830+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24640035b0 con 0x7f247c0b4090 2023-11-24T06:18:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3560136542 0 0) 0x7f24640035b0 con 0x7f247c0b4090 2023-11-24T06:18:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f247c15d6a0 msgr2=0x7f247c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f247c15d6a0 0x7f247c0b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 >> v1:172.21.15.123:6789/0 conn(0x7f247c071be0 legacy=0x7f247c140fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c0726a0 con 0x7f247c0b4090 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f246c001670 con 0x7f247c0b4090 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 >> v1:172.21.15.89:6789/0 conn(0x7f247c0b4090 legacy=0x7f247c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24640037f0 0x7f2464003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f246400b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f246400b630 0x7f246400daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c072d60 con 0x7f2464004130 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246400e380 con 0x7f246400b630 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246400e540 con 0x7f24640037f0 2023-11-24T06:18:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.831+0000 7f247a7fc700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f246400b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.832+0000 7f247a7fc700 1 -- 172.21.15.89:0/1248820258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24640037f0 msgr2=0x7f2464003bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:18:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.832+0000 7f247a7fc700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24640037f0 0x7f2464003bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.832+0000 7f247a7fc700 1 -- 172.21.15.89:0/1248820258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f246400b630 msgr2=0x7f246400daa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:18:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.832+0000 7f247a7fc700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f246400b630 0x7f246400daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.832+0000 7f247a7fc700 1 -- 172.21.15.89:0/1248820258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c0726a0 con 0x7f2464004130 2023-11-24T06:18:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.832+0000 7f247a7fc700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f246400b0a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f246c007d80 tx=0x7f2464004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4f86465eb6ab7d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.833+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c00ce80 con 0x7f2464004130 2023-11-24T06:18:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.833+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c005960 con 0x7f2464004130 2023-11-24T06:18:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.833+0000 7f2479ffb700 1 -- 172.21.15.89:0/1248820258 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c005cd0 con 0x7f2464004130 2023-11-24T06:18:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.834+0000 7f24817f2700 1 -- 172.21.15.89:0/1248820258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 msgr2=0x7f246400b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.834+0000 7f24817f2700 1 --2- 172.21.15.89:0/1248820258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f246400b0a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f246c007d80 tx=0x7f2464004870 comp rx=0 tx=0).stop 2023-11-24T06:18:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.834+0000 7f247b7fe700 1 -- 172.21.15.89:0/1248820258 reap_dead start 2023-11-24T06:18:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.835+0000 7f24817f2700 1 -- 172.21.15.89:0/1248820258 shutdown_connections 2023-11-24T06:18:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.835+0000 7f24817f2700 1 -- 172.21.15.89:0/1248820258 >> 172.21.15.89:0/1248820258 conn(0x7f247c0cfa30 msgr2=0x7f247c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.835+0000 7f24817f2700 1 -- 172.21.15.89:0/1248820258 shutdown_connections 2023-11-24T06:18:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.835+0000 7f24817f2700 1 -- 172.21.15.89:0/1248820258 wait complete. 2023-11-24T06:18:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.837+0000 7f24817f2700 1 Processor -- start 2023-11-24T06:18:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.837+0000 7f24817f2700 1 -- start start 2023-11-24T06:18:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.839+0000 7f24817f2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f247c0aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.839+0000 7f24817f2700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f247c0ce0f0 0x7f247c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.839+0000 7f24817f2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f247c0ce570 0x7f247c15b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.839+0000 7f24817f2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0a9460 con 0x7f2464004130 2023-11-24T06:18:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.839+0000 7f24817f2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0a9670 con 0x7f247c0ce570 2023-11-24T06:18:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.839+0000 7f24817f2700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0a9950 con 0x7f247c0ce0f0 2023-11-24T06:18:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.839+0000 7f247affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f247c0aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.840+0000 7f247affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f247c0aad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35362/0 (socket says 172.21.15.89:35362) 2023-11-24T06:18:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.840+0000 7f247affd700 1 -- 172.21.15.89:0/1881501886 learned_addr learned my addr 172.21.15.89:0/1881501886 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.840+0000 7f247affd700 1 -- 172.21.15.89:0/1881501886 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f247c0ce0f0 msgr2=0x7f247c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.840+0000 7f247affd700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f247c0ce0f0 0x7f247c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.840+0000 7f247affd700 1 -- 172.21.15.89:0/1881501886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f247c0ce570 msgr2=0x7f247c15b880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-24T06:18:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.840+0000 7f247affd700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f247c0ce570 0x7f247c15b880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.840+0000 7f247affd700 1 -- 172.21.15.89:0/1881501886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c0a9c30 con 0x7f2464004130 2023-11-24T06:18:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.841+0000 7f247affd700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f247c0aad70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f247400ccd0 tx=0x7f247400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.841+0000 7f246b7fe700 1 -- 172.21.15.89:0/1881501886 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247400ce70 con 0x7f2464004130 2023-11-24T06:18:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.841+0000 7f246b7fe700 1 -- 172.21.15.89:0/1881501886 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2474003a20 con 0x7f2464004130 2023-11-24T06:18:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.841+0000 7f246b7fe700 1 -- 172.21.15.89:0/1881501886 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2474003dd0 con 0x7f2464004130 2023-11-24T06:18:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.842+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247c0a9f10 con 0x7f2464004130 2023-11-24T06:18:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.842+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c0ab2b0 con 0x7f2464004130 2023-11-24T06:18:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.847+0000 7f246b7fe700 1 -- 172.21.15.89:0/1881501886 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2474004020 con 0x7f2464004130 2023-11-24T06:18:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.847+0000 7f246b7fe700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f246406cb80 0x7f246406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.848+0000 7f246b7fe700 1 -- 172.21.15.89:0/1881501886 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2474012070 con 0x7f2464004130 2023-11-24T06:18:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.848+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24580052e0 con 0x7f2464004130 2023-11-24T06:18:57.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.857+0000 7f246b7fe700 1 -- 172.21.15.89:0/1881501886 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2474054ca0 con 0x7f2464004130 2023-11-24T06:18:57.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.859+0000 7f247a7fc700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f246406cb80 0x7f246406f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.860+0000 7f247a7fc700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f246406cb80 0x7f246406f040 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f246c000c00 tx=0x7f246c013000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.886+0000 7f6f4fc14700 1 Processor -- start 2023-11-24T06:18:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.886+0000 7f6f4fc14700 1 -- start start 2023-11-24T06:18:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.888+0000 7f6f4fc14700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f4815d6a0 0x7f6f480b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.888+0000 7f6f4fc14700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f48071be0 0x7f6f48140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.888+0000 7f6f4fc14700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f480720e0 con 0x7f6f480b4090 2023-11-24T06:18:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.888+0000 7f6f4fc14700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f48072d60 con 0x7f6f48071be0 2023-11-24T06:18:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.888+0000 7f6f4fc14700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4806de80 con 0x7f6f4815d6a0 2023-11-24T06:18:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.889+0000 7f6f4d9b0700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f4815d6a0 0x7f6f480b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.889+0000 7f6f4d9b0700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f4815d6a0 0x7f6f480b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50482/0 (socket says 172.21.15.89:50482) 2023-11-24T06:18:57.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.889+0000 7f6f4d9b0700 1 -- 172.21.15.89:0/1146563066 learned_addr learned my addr 172.21.15.89:0/1146563066 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.889+0000 7f6f4c9ae700 1 -- 172.21.15.89:0/1146563066 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335143654 0 0) 0x7f6f480720e0 con 0x7f6f480b4090 2023-11-24T06:18:57.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4c9ae700 1 -- 172.21.15.89:0/1146563066 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f340035b0 con 0x7f6f480b4090 2023-11-24T06:18:57.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4d9b0700 1 -- 172.21.15.89:0/1146563066 >> v2:172.21.15.89:3300/0 conn(0x7f6f48071be0 msgr2=0x7f6f48140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:57.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4d9b0700 1 --2- 172.21.15.89:0/1146563066 >> v2:172.21.15.89:3300/0 conn(0x7f6f48071be0 0x7f6f48140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4d9b0700 1 -- 172.21.15.89:0/1146563066 >> v1:172.21.15.123:6789/0 conn(0x7f6f480b4090 legacy=0x7f6f480714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4d9b0700 1 -- 172.21.15.89:0/1146563066 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f480723c0 con 0x7f6f4815d6a0 2023-11-24T06:18:57.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4d9b0700 1 --2- 172.21.15.89:0/1146563066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 0x7f6f480b3b50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6f480cfa00 tx=0x7f6f4000d650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3196aca67019500 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4c9ae700 1 -- 172.21.15.89:0/1146563066 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40010070 con 0x7f6f4815d6a0 2023-11-24T06:18:57.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4c9ae700 1 -- 172.21.15.89:0/1146563066 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4000dc30 con 0x7f6f4815d6a0 2023-11-24T06:18:57.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.890+0000 7f6f4c9ae700 1 -- 172.21.15.89:0/1146563066 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40007b30 con 0x7f6f4815d6a0 2023-11-24T06:18:57.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.891+0000 7f6f4fc14700 1 -- 172.21.15.89:0/1146563066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 msgr2=0x7f6f480b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.891+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/1146563066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 0x7f6f480b3b50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6f480cfa00 tx=0x7f6f4000d650 comp rx=0 tx=0).stop 2023-11-24T06:18:57.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.891+0000 7f6f4fc14700 1 -- 172.21.15.89:0/1146563066 shutdown_connections 2023-11-24T06:18:57.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.891+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/1146563066 >> v2:172.21.15.89:3300/0 conn(0x7f6f48071be0 0x7f6f48140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.891+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/1146563066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 0x7f6f480b3b50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.891+0000 7f6f4fc14700 1 -- 172.21.15.89:0/1146563066 >> 172.21.15.89:0/1146563066 conn(0x7f6f480cfa30 msgr2=0x7f6f480cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:57.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.892+0000 7f6f4fc14700 1 -- 172.21.15.89:0/1146563066 shutdown_connections 2023-11-24T06:18:57.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.892+0000 7f6f4fc14700 1 -- 172.21.15.89:0/1146563066 wait complete. 2023-11-24T06:18:57.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.893+0000 7f6f4fc14700 1 Processor -- start 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.893+0000 7f6f4fc14700 1 -- start start 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.894+0000 7f6f4fc14700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f48071be0 0x7f6f481fbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.894+0000 7f6f4fc14700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f480b4090 0x7f6f481f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.895+0000 7f6f4fc14700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 0x7f6f480aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.895+0000 7f6f4fc14700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481fc360 con 0x7f6f480b4090 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.895+0000 7f6f4fc14700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480cd470 con 0x7f6f4815d6a0 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.895+0000 7f6f4fc14700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481f8370 con 0x7f6f48071be0 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.895+0000 7f6f4e1b1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 0x7f6f480aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.895+0000 7f6f4e1b1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 0x7f6f480aa680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50488/0 (socket says 172.21.15.89:50488) 2023-11-24T06:18:57.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.895+0000 7f6f4e1b1700 1 -- 172.21.15.89:0/694454641 learned_addr learned my addr 172.21.15.89:0/694454641 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.895+0000 7f6f4d9b0700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f48071be0 0x7f6f481fbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f4d9b0700 1 -- 172.21.15.89:0/694454641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 msgr2=0x7f6f480aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f4d9b0700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 0x7f6f480aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f4d9b0700 1 -- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f480b4090 msgr2=0x7f6f481f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f4d9b0700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f480b4090 0x7f6f481f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f4d9b0700 1 -- 172.21.15.89:0/694454641 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4000d050 con 0x7f6f48071be0 2023-11-24T06:18:57.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f4d9b0700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f48071be0 0x7f6f481fbe20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6f40017000 tx=0x7f6f40004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f3e7fc700 1 -- 172.21.15.89:0/694454641 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40004500 con 0x7f6f48071be0 2023-11-24T06:18:57.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f3e7fc700 1 -- 172.21.15.89:0/694454641 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f40004680 con 0x7f6f48071be0 2023-11-24T06:18:57.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f3e7fc700 1 -- 172.21.15.89:0/694454641 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40008040 con 0x7f6f48071be0 2023-11-24T06:18:57.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f481f8650 con 0x7f6f48071be0 2023-11-24T06:18:57.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f481f8c10 con 0x7f6f48071be0 2023-11-24T06:18:57.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.897+0000 7fef66357700 1 Processor -- start 2023-11-24T06:18:57.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.898+0000 7fef66357700 1 -- start start 2023-11-24T06:18:57.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7fef66357700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fef60164a90 0x7fef6006eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7fef66357700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef6006d4d0 0x7fef6016bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7fef66357700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6006ced0 con 0x7fef600b3b50 2023-11-24T06:18:57.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7fef66357700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6006da70 con 0x7fef6006d4d0 2023-11-24T06:18:57.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7fef66357700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef601634e0 con 0x7fef60164a90 2023-11-24T06:18:57.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f300052e0 con 0x7f6f48071be0 2023-11-24T06:18:57.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7f333aaff700 1 Processor -- start 2023-11-24T06:18:57.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7fef648f4700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef6006d4d0 0x7fef6016bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7fef648f4700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef6006d4d0 0x7fef6016bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35384/0 (socket says 172.21.15.89:35384) 2023-11-24T06:18:57.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7fef648f4700 1 -- 172.21.15.89:0/591050148 learned_addr learned my addr 172.21.15.89:0/591050148 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.900+0000 7fef648f4700 1 -- 172.21.15.89:0/591050148 >> v2:172.21.15.123:3300/0 conn(0x7fef60164a90 msgr2=0x7fef6006eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:57.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.900+0000 7fef648f4700 1 --2- 172.21.15.89:0/591050148 >> v2:172.21.15.123:3300/0 conn(0x7fef60164a90 0x7fef6006eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.900+0000 7fef648f4700 1 -- 172.21.15.89:0/591050148 >> v1:172.21.15.123:6789/0 conn(0x7fef600b3b50 legacy=0x7fef600b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:57.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.900+0000 7fef648f4700 1 -- 172.21.15.89:0/591050148 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef601637c0 con 0x7fef6006d4d0 2023-11-24T06:18:57.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.900+0000 7fef648f4700 1 --2- 172.21.15.89:0/591050148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef6006d4d0 0x7fef6016bee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef60062350 tx=0x7fef48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848b91fabc0db569 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.900+0000 7fef5effd700 1 -- 172.21.15.89:0/591050148 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48004230 con 0x7fef6006d4d0 2023-11-24T06:18:57.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.900+0000 7fef5effd700 1 -- 172.21.15.89:0/591050148 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef48004390 con 0x7fef6006d4d0 2023-11-24T06:18:57.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.900+0000 7fef5effd700 1 -- 172.21.15.89:0/591050148 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef480036a0 con 0x7fef6006d4d0 2023-11-24T06:18:57.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f8796979700 1 Processor -- start 2023-11-24T06:18:57.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f8796979700 1 -- start start 2023-11-24T06:18:57.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7fef66357700 1 -- 172.21.15.89:0/591050148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef6006d4d0 msgr2=0x7fef6016bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7fef66357700 1 --2- 172.21.15.89:0/591050148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef6006d4d0 0x7fef6016bee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef60062350 tx=0x7fef48009bd0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7fef66357700 1 -- 172.21.15.89:0/591050148 shutdown_connections 2023-11-24T06:18:57.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7fef66357700 1 --2- 172.21.15.89:0/591050148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef6006d4d0 0x7fef6016bee0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7fef66357700 1 --2- 172.21.15.89:0/591050148 >> v2:172.21.15.123:3300/0 conn(0x7fef60164a90 0x7fef6006eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7fef66357700 1 -- 172.21.15.89:0/591050148 >> 172.21.15.89:0/591050148 conn(0x7fef600cfa30 msgr2=0x7fef600cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:57.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7fef66357700 1 -- 172.21.15.89:0/591050148 shutdown_connections 2023-11-24T06:18:57.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.899+0000 7f333aaff700 1 -- start start 2023-11-24T06:18:57.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f333aaff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f33340cf440 0x7f333406e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f333aaff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3334070fd0 0x7f3334165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f333aaff700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334072110 con 0x7f33341663d0 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f333aaff700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3334166860 con 0x7f3334070fd0 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f333aaff700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f333406ee00 con 0x7f33340cf440 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f3333fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3334070fd0 0x7f3334165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f3333fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3334070fd0 0x7f3334165e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35388/0 (socket says 172.21.15.89:35388) 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.901+0000 7f3333fff700 1 -- 172.21.15.89:0/3303761866 learned_addr learned my addr 172.21.15.89:0/3303761866 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f3333fff700 1 -- 172.21.15.89:0/3303761866 >> v2:172.21.15.123:3300/0 conn(0x7f33340cf440 msgr2=0x7f333406e8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f3333fff700 1 --2- 172.21.15.89:0/3303761866 >> v2:172.21.15.123:3300/0 conn(0x7f33340cf440 0x7f333406e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f3333fff700 1 -- 172.21.15.89:0/3303761866 >> v1:172.21.15.123:6789/0 conn(0x7f33341663d0 legacy=0x7f33341628a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f3333fff700 1 -- 172.21.15.89:0/3303761866 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334071500 con 0x7f3334070fd0 2023-11-24T06:18:57.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f3333fff700 1 --2- 172.21.15.89:0/3303761866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 0x7f3334165e90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f33340b5cb0 tx=0x7f3328009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a103d8b70304fc40 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f8796979700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879014be10 0x7f879006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f8796979700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f87900b4260 0x7f8790071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f8796979700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87900714e0 con 0x7f879006efc0 2023-11-24T06:18:57.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f8796979700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f879006f440 con 0x7f87900b4260 2023-11-24T06:18:57.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f8796979700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879014c1f0 con 0x7f879014be10 2023-11-24T06:18:57.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f33337fe700 1 -- 172.21.15.89:0/3303761866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3328004230 con 0x7f3334070fd0 2023-11-24T06:18:57.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f33337fe700 1 -- 172.21.15.89:0/3303761866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3328004390 con 0x7f3334070fd0 2023-11-24T06:18:57.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f33337fe700 1 -- 172.21.15.89:0/3303761866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33280036a0 con 0x7f3334070fd0 2023-11-24T06:18:57.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f6f3e7fc700 1 -- 172.21.15.89:0/694454641 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6f400081a0 con 0x7f6f48071be0 2023-11-24T06:18:57.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f6f3e7fc700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6f2c06afa0 0x7f6f2c06d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7f6f3e7fc700 1 -- 172.21.15.89:0/694454641 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f4008b280 con 0x7f6f48071be0 2023-11-24T06:18:57.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f878ffff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879014be10 0x7f879006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f878ffff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879014be10 0x7f879006d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44268/0 (socket says 172.21.15.89:44268) 2023-11-24T06:18:57.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f878ffff700 1 -- 172.21.15.89:0/4011266107 learned_addr learned my addr 172.21.15.89:0/4011266107 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.902+0000 7fef66357700 1 -- 172.21.15.89:0/591050148 wait complete. 2023-11-24T06:18:57.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f6f4d1af700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6f2c06afa0 0x7f6f2c06d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f878ffff700 1 -- 172.21.15.89:0/4011266107 >> v2:172.21.15.123:3300/0 conn(0x7f87900b4260 msgr2=0x7f8790071970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:57.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f878ffff700 1 --2- 172.21.15.89:0/4011266107 >> v2:172.21.15.123:3300/0 conn(0x7f87900b4260 0x7f8790071970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f878ffff700 1 -- 172.21.15.89:0/4011266107 >> v1:172.21.15.123:6789/0 conn(0x7f879006efc0 legacy=0x7f87900b3b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:18:57.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f878ffff700 1 -- 172.21.15.89:0/4011266107 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8790070bf0 con 0x7f879014be10 2023-11-24T06:18:57.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f333aaff700 1 -- 172.21.15.89:0/3303761866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 msgr2=0x7f3334165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f333aaff700 1 --2- 172.21.15.89:0/3303761866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 0x7f3334165e90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f33340b5cb0 tx=0x7f3328009bd0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f333aaff700 1 -- 172.21.15.89:0/3303761866 shutdown_connections 2023-11-24T06:18:57.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f333aaff700 1 --2- 172.21.15.89:0/3303761866 >> v2:172.21.15.123:3300/0 conn(0x7f33340cf440 0x7f333406e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f333aaff700 1 --2- 172.21.15.89:0/3303761866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 0x7f3334165e90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f333aaff700 1 -- 172.21.15.89:0/3303761866 >> 172.21.15.89:0/3303761866 conn(0x7f33340b1740 msgr2=0x7f33340b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:57.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f333aaff700 1 -- 172.21.15.89:0/3303761866 shutdown_connections 2023-11-24T06:18:57.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f8717ebd700 1 Processor -- start 2023-11-24T06:18:57.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.903+0000 7f878ffff700 1 --2- 172.21.15.89:0/4011266107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879014be10 0x7f879006d4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8788009e50 tx=0x7f878800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61dfe2c0dbc1b9c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7fef66357700 1 Processor -- start 2023-11-24T06:18:57.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f6f4d1af700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6f2c06afa0 0x7f6f2c06d460 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f440010e0 tx=0x7f6f440079e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f8717ebd700 1 -- start start 2023-11-24T06:18:57.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7fef66357700 1 -- start start 2023-11-24T06:18:57.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.905+0000 7fef66357700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef6006d4d0 0x7fef601388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.905+0000 7fef66357700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef600b3b50 0x7fef600cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.905+0000 7fef66357700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef60164a90 0x7fef6014a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.905+0000 7fef66357700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600ce780 con 0x7fef60164a90 2023-11-24T06:18:57.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.905+0000 7fef66357700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef60138eb0 con 0x7fef6006d4d0 2023-11-24T06:18:57.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.905+0000 7fef66357700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef60139190 con 0x7fef600b3b50 2023-11-24T06:18:57.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef6006d4d0 0x7fef601388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef6006d4d0 0x7fef601388d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50510/0 (socket says 172.21.15.89:50510) 2023-11-24T06:18:57.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 -- 172.21.15.89:0/3629446841 learned_addr learned my addr 172.21.15.89:0/3629446841 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef648f4700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef60164a90 0x7fef6014a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f8717ebd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f87100ae4e0 0x7f871006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f8717ebd700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87100b46e0 con 0x7f871006f360 2023-11-24T06:18:57.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f8717ebd700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87100b49c0 con 0x7f8710162240 2023-11-24T06:18:57.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f8717ebd700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87100b4ca0 con 0x7f87100ae4e0 2023-11-24T06:18:57.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f09bd9af700 1 Processor -- start 2023-11-24T06:18:57.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 -- 172.21.15.89:0/3629446841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef600b3b50 msgr2=0x7fef600cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef600b3b50 0x7fef600cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 -- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef60164a90 msgr2=0x7fef6014a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef60164a90 0x7fef6014a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 -- 172.21.15.89:0/3629446841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef48009550 con 0x7fef6006d4d0 2023-11-24T06:18:57.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef648f4700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef60164a90 0x7fef6014a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:18:57.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7fef5ffff700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef6006d4d0 0x7fef601388d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fef5800c8c0 tx=0x7fef5800cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.904+0000 7f333aaff700 1 -- 172.21.15.89:0/3303761866 wait complete. 2023-11-24T06:18:57.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f333aaff700 1 Processor -- start 2023-11-24T06:18:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f333aaff700 1 -- start start 2023-11-24T06:18:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333aaff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 0x7f333417ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333aaff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf440 0x7f33340b0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333aaff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33341663d0 0x7f333414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333aaff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333412eee0 con 0x7f3334070fd0 2023-11-24T06:18:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333aaff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333412dea0 con 0x7f33341663d0 2023-11-24T06:18:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333aaff700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333412e120 con 0x7f33340cf440 2023-11-24T06:18:57.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f09bd9af700 1 -- start start 2023-11-24T06:18:57.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f871645a700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f871006f360 0x7f8710165f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38908/0 (socket says 172.21.15.89:38908) 2023-11-24T06:18:57.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f871645a700 1 -- 172.21.15.89:0/1124223610 learned_addr learned my addr 172.21.15.89:0/1124223610 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f8715458700 1 --2- 172.21.15.89:0/1124223610 >> v2:172.21.15.123:3300/0 conn(0x7f87100ae4e0 0x7f871006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.906+0000 7f8714c57700 1 -- 172.21.15.89:0/1124223610 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306314313 0 0) 0x7f87100b46e0 con 0x7f871006f360 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8714c57700 1 -- 172.21.15.89:0/1124223610 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc0035b0 con 0x7f871006f360 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8714c57700 1 -- 172.21.15.89:0/1124223610 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573351919 0 0) 0x7f87100b49c0 con 0x7f8710162240 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8714c57700 1 -- 172.21.15.89:0/1124223610 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87100b46e0 con 0x7f8710162240 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8715458700 1 -- 172.21.15.89:0/1124223610 >> v1:172.21.15.123:6789/0 conn(0x7f8710162240 legacy=0x7f87100afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8715458700 1 -- 172.21.15.89:0/1124223610 >> v1:172.21.15.89:6789/0 conn(0x7f871006f360 legacy=0x7f8710165f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8715458700 1 -- 172.21.15.89:0/1124223610 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87100b4f80 con 0x7f87100ae4e0 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8715458700 1 --2- 172.21.15.89:0/1124223610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87100ae4e0 0x7f871006c000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f870c009840 tx=0x7f870c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80a4fafb9c218eb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8714c57700 1 -- 172.21.15.89:0/1124223610 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c011830 con 0x7f87100ae4e0 2023-11-24T06:18:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8714c57700 1 -- 172.21.15.89:0/1124223610 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c011990 con 0x7f87100ae4e0 2023-11-24T06:18:57.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f8714c57700 1 -- 172.21.15.89:0/1124223610 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c011d20 con 0x7f87100ae4e0 2023-11-24T06:18:57.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333889b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 0x7f333417ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333889b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 0x7f333417ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35410/0 (socket says 172.21.15.89:35410) 2023-11-24T06:18:57.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333889b700 1 -- 172.21.15.89:0/1987188901 learned_addr learned my addr 172.21.15.89:0/1987188901 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f333909c700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33341663d0 0x7f333414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f09bd9af700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f09b813f170 0x7f09b806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f09bd9af700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b806c540 0x7f09b815ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f09bd9af700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b806c9a0 con 0x7f09b8160440 2023-11-24T06:18:57.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f09bd9af700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b815c5b0 con 0x7f09b813f170 2023-11-24T06:18:57.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f09bd9af700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b815c880 con 0x7f09b806c540 2023-11-24T06:18:57.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7fef5cff9700 1 -- 172.21.15.89:0/3629446841 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58003e10 con 0x7fef6006d4d0 2023-11-24T06:18:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7fef5cff9700 1 -- 172.21.15.89:0/3629446841 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef58005070 con 0x7fef6006d4d0 2023-11-24T06:18:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7fef5cff9700 1 -- 172.21.15.89:0/3629446841 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58005420 con 0x7fef6006d4d0 2023-11-24T06:18:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef60139470 con 0x7fef6006d4d0 2023-11-24T06:18:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef60155870 con 0x7fef6006d4d0 2023-11-24T06:18:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef600ce550 con 0x7fef6006d4d0 2023-11-24T06:18:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.907+0000 7f878effd700 1 -- 172.21.15.89:0/4011266107 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878800e070 con 0x7f879014be10 2023-11-24T06:18:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f878effd700 1 -- 172.21.15.89:0/4011266107 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8788003a20 con 0x7f879014be10 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f878effd700 1 -- 172.21.15.89:0/4011266107 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788003dd0 con 0x7f879014be10 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8796979700 1 -- 172.21.15.89:0/4011266107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879014be10 msgr2=0x7f879006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8796979700 1 --2- 172.21.15.89:0/4011266107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879014be10 0x7f879006d4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8788009e50 tx=0x7f878800b330 comp rx=0 tx=0).stop 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8796979700 1 -- 172.21.15.89:0/4011266107 shutdown_connections 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8796979700 1 --2- 172.21.15.89:0/4011266107 >> v2:172.21.15.123:3300/0 conn(0x7f87900b4260 0x7f8790071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8796979700 1 --2- 172.21.15.89:0/4011266107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879014be10 0x7f879006d4d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f8796979700 1 -- 172.21.15.89:0/4011266107 >> 172.21.15.89:0/4011266107 conn(0x7f87900cfa30 msgr2=0x7f87900cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f333909c700 1 -- 172.21.15.89:0/1987188901 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf440 msgr2=0x7f33340b0950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:18:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f333909c700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf440 0x7f33340b0950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f333909c700 1 -- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 msgr2=0x7f333417ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f333909c700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 0x7f333417ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f333909c700 1 -- 172.21.15.89:0/1987188901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3328009550 con 0x7f33341663d0 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f333909c700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33341663d0 0x7f333414d060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f332c00be20 tx=0x7f332c009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f09b67fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b806c540 0x7f09b815ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f09b67fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b806c540 0x7f09b815ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44292/0 (socket says 172.21.15.89:44292) 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f09b67fc700 1 -- 172.21.15.89:0/3870404112 learned_addr learned my addr 172.21.15.89:0/3870404112 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f09b67fc700 1 -- 172.21.15.89:0/3870404112 >> v2:172.21.15.89:3300/0 conn(0x7f09b813f170 msgr2=0x7f09b806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f09b67fc700 1 --2- 172.21.15.89:0/3870404112 >> v2:172.21.15.89:3300/0 conn(0x7f09b813f170 0x7f09b806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f09b67fc700 1 -- 172.21.15.89:0/3870404112 >> v1:172.21.15.89:6789/0 conn(0x7f09b8160440 legacy=0x7f09b8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f09b67fc700 1 -- 172.21.15.89:0/3870404112 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b815cb60 con 0x7f09b806c540 2023-11-24T06:18:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8717ebd700 1 -- 172.21.15.89:0/1124223610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87100ae4e0 msgr2=0x7f871006c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8717ebd700 1 --2- 172.21.15.89:0/1124223610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87100ae4e0 0x7f871006c000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f870c009840 tx=0x7f870c009b50 comp rx=0 tx=0).stop 2023-11-24T06:18:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8717ebd700 1 -- 172.21.15.89:0/1124223610 shutdown_connections 2023-11-24T06:18:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8717ebd700 1 --2- 172.21.15.89:0/1124223610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87100ae4e0 0x7f871006c000 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f870c009840 tx=0x7f870c009b50 comp rx=0 tx=0).stop 2023-11-24T06:18:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.908+0000 7f8717ebd700 1 -- 172.21.15.89:0/1124223610 >> 172.21.15.89:0/1124223610 conn(0x7f8710071b90 msgr2=0x7f8710071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:57.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f8796979700 1 -- 172.21.15.89:0/4011266107 shutdown_connections 2023-11-24T06:18:57.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f6f3e7fc700 1 -- 172.21.15.89:0/694454641 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f400564c0 con 0x7f6f48071be0 2023-11-24T06:18:57.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f09b67fc700 1 --2- 172.21.15.89:0/3870404112 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b806c540 0x7f09b815ff00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f09a80098f0 tx=0x7f09a8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6340b8b2cf3a6926 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f8717ebd700 1 -- 172.21.15.89:0/1124223610 shutdown_connections 2023-11-24T06:18:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f09b5ffb700 1 -- 172.21.15.89:0/3870404112 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8004230 con 0x7f09b806c540 2023-11-24T06:18:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f09b5ffb700 1 -- 172.21.15.89:0/3870404112 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8004390 con 0x7f09b806c540 2023-11-24T06:18:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f09b5ffb700 1 -- 172.21.15.89:0/3870404112 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a80036a0 con 0x7f09b806c540 2023-11-24T06:18:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f8717ebd700 1 -- 172.21.15.89:0/1124223610 wait complete. 2023-11-24T06:18:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f33317fa700 1 -- 172.21.15.89:0/1987188901 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c00cc90 con 0x7f33341663d0 2023-11-24T06:18:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f8796979700 1 -- 172.21.15.89:0/4011266107 wait complete. 2023-11-24T06:18:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.910+0000 7f09bd9af700 1 -- 172.21.15.89:0/3870404112 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b806c540 msgr2=0x7f09b815ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.910+0000 7f09bd9af700 1 --2- 172.21.15.89:0/3870404112 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b806c540 0x7f09b815ff00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f09a80098f0 tx=0x7f09a8009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f33317fa700 1 -- 172.21.15.89:0/1987188901 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332c00ce10 con 0x7f33341663d0 2023-11-24T06:18:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f33317fa700 1 -- 172.21.15.89:0/1987188901 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c014680 con 0x7f33341663d0 2023-11-24T06:18:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.909+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333412e400 con 0x7f33341663d0 2023-11-24T06:18:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.910+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333412e9c0 con 0x7f33341663d0 2023-11-24T06:18:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.910+0000 7f8796979700 1 Processor -- start 2023-11-24T06:18:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.910+0000 7f8796979700 1 -- start start 2023-11-24T06:18:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.910+0000 7f8717ebd700 1 Processor -- start 2023-11-24T06:18:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f8717ebd700 1 -- start start 2023-11-24T06:18:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333412ebe0 con 0x7f33341663d0 2023-11-24T06:18:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f09bd9af700 1 -- 172.21.15.89:0/3870404112 shutdown_connections 2023-11-24T06:18:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f09bd9af700 1 --2- 172.21.15.89:0/3870404112 >> v2:172.21.15.89:3300/0 conn(0x7f09b813f170 0x7f09b806c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f09bd9af700 1 --2- 172.21.15.89:0/3870404112 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b806c540 0x7f09b815ff00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f09bd9af700 1 -- 172.21.15.89:0/3870404112 >> 172.21.15.89:0/3870404112 conn(0x7f09b8071ac0 msgr2=0x7f09b8071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f8796979700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879006efc0 0x7f8790161d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f8796979700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900b4260 0x7f8790143e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f8796979700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879014be10 0x7f87901fbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f8796979700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87900cbe30 con 0x7f879014be10 2023-11-24T06:18:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f8796979700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87901f9b20 con 0x7f87900b4260 2023-11-24T06:18:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f8796979700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87901f9e00 con 0x7f879006efc0 2023-11-24T06:18:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f09bd9af700 1 -- 172.21.15.89:0/3870404112 shutdown_connections 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8717ebd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871006f360 0x7f871015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8717ebd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8710162240 0x7f87101634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8717ebd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87101593f0 0x7f871014b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8717ebd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101f9140 con 0x7f871006f360 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8717ebd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101f84c0 con 0x7f87101593f0 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8717ebd700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87101f87a0 con 0x7f8710162240 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8715458700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8710162240 0x7f87101634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8715458700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8710162240 0x7f87101634a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44306/0 (socket says 172.21.15.89:44306) 2023-11-24T06:18:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8715458700 1 -- 172.21.15.89:0/591173846 learned_addr learned my addr 172.21.15.89:0/591173846 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7fef5cff9700 1 -- 172.21.15.89:0/3629446841 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fef58005580 con 0x7fef6006d4d0 2023-11-24T06:18:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f09bd9af700 1 -- 172.21.15.89:0/3870404112 wait complete. 2023-11-24T06:18:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f09bd9af700 1 Processor -- start 2023-11-24T06:18:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7fef5cff9700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fef4006af20 0x7fef4006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f878ffff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879006efc0 0x7f8790161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f8794f16700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879014be10 0x7f87901fbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.911+0000 7f878ffff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879006efc0 0x7f8790161d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44298/0 (socket says 172.21.15.89:44298) 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878ffff700 1 -- 172.21.15.89:0/263040466 learned_addr learned my addr 172.21.15.89:0/263040466 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878f7fe700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900b4260 0x7f8790143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878ffff700 1 -- 172.21.15.89:0/263040466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900b4260 msgr2=0x7f8790143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878ffff700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900b4260 0x7f8790143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878ffff700 1 -- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879014be10 msgr2=0x7f87901fbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878ffff700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879014be10 0x7f87901fbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878ffff700 1 -- 172.21.15.89:0/263040466 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788009b30 con 0x7f879006efc0 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878ffff700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879006efc0 0x7f8790161d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f879014c330 tx=0x7f8788004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878cff9700 1 -- 172.21.15.89:0/263040466 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788004440 con 0x7f879006efc0 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878cff9700 1 -- 172.21.15.89:0/263040466 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87880092e0 con 0x7f879006efc0 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f878cff9700 1 -- 172.21.15.89:0/263040466 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87880047e0 con 0x7f879006efc0 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87901fa0e0 con 0x7f879006efc0 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.912+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87901fa6a0 con 0x7f879006efc0 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f8715458700 1 -- 172.21.15.89:0/591173846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87101593f0 msgr2=0x7f871014b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f8715458700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87101593f0 0x7f871014b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f8715458700 1 -- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871006f360 msgr2=0x7f871015f0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-24T06:18:57.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f8715458700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871006f360 0x7f871015f0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f8715458700 1 -- 172.21.15.89:0/591173846 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c0094a0 con 0x7f8710162240 2023-11-24T06:18:57.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f8715458700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8710162240 0x7f87101634a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f870c0090b0 tx=0x7f870c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f87067fc700 1 -- 172.21.15.89:0/591173846 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c01a9b0 con 0x7f8710162240 2023-11-24T06:18:57.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7fef5cff9700 1 -- 172.21.15.89:0/3629446841 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef580890a0 con 0x7fef6006d4d0 2023-11-24T06:18:57.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7fef5cff9700 1 -- 172.21.15.89:0/3629446841 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef58054130 con 0x7fef6006d4d0 2023-11-24T06:18:57.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87900af530 con 0x7f879006efc0 2023-11-24T06:18:57.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f878cff9700 1 -- 172.21.15.89:0/263040466 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8788012430 con 0x7f879006efc0 2023-11-24T06:18:57.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f09bd9af700 1 -- start start 2023-11-24T06:18:57.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f09bd9af700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b806c540 0x7f09b8152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f09bd9af700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b813f170 0x7f09b8143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f09bd9af700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b8160440 0x7f09b81442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f09bd9af700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8152da0 con 0x7f09b806c540 2023-11-24T06:18:57.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f09bd9af700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b813fb80 con 0x7f09b813f170 2023-11-24T06:18:57.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f09bd9af700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8137950 con 0x7f09b8160440 2023-11-24T06:18:57.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87101f8a80 con 0x7f8710162240 2023-11-24T06:18:57.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f87067fc700 1 -- 172.21.15.89:0/591173846 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c01ab10 con 0x7f8710162240 2023-11-24T06:18:57.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87101f8140 con 0x7f8710162240 2023-11-24T06:18:57.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.913+0000 7f87067fc700 1 -- 172.21.15.89:0/591173846 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c01ae80 con 0x7f8710162240 2023-11-24T06:18:57.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b77fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b8160440 0x7f09b81442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b67fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b813f170 0x7f09b8143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b77fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b8160440 0x7f09b81442c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44314/0 (socket says 172.21.15.89:44314) 2023-11-24T06:18:57.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b77fe700 1 -- 172.21.15.89:0/917902014 learned_addr learned my addr 172.21.15.89:0/917902014 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b67fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b813f170 0x7f09b8143d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50548/0 (socket says 172.21.15.89:50548) 2023-11-24T06:18:57.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f878cff9700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f878cff9700 1 -- 172.21.15.89:0/263040466 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f878808ac10 con 0x7f879006efc0 2023-11-24T06:18:57.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f878f7fe700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b67fc700 1 -- 172.21.15.89:0/917902014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b8160440 msgr2=0x7f09b81442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b67fc700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b8160440 0x7f09b81442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b67fc700 1 -- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b806c540 msgr2=0x7f09b8152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b67fc700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b806c540 0x7f09b8152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b67fc700 1 -- 172.21.15.89:0/917902014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a8009550 con 0x7f09b813f170 2023-11-24T06:18:57.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.914+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87100753f0 con 0x7f8710162240 2023-11-24T06:18:57.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.915+0000 7f09b67fc700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b813f170 0x7f09b8143d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09a8015000 tx=0x7f09a8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.916+0000 7f878f7fe700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8790148000 tx=0x7f8780009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.916+0000 7f33317fa700 1 -- 172.21.15.89:0/1987188901 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f332c0147e0 con 0x7f33341663d0 2023-11-24T06:18:57.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.916+0000 7f33317fa700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f332006af20 0x7f332006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.916+0000 7f333889b700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f332006af20 0x7f332006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.916+0000 7f33317fa700 1 -- 172.21.15.89:0/1987188901 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f332c089b80 con 0x7f33341663d0 2023-11-24T06:18:57.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.917+0000 7f333889b700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f332006af20 0x7f332006d3e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3334145910 tx=0x7f3324009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.919+0000 7f099ffff700 1 -- 172.21.15.89:0/917902014 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8003dc0 con 0x7f09b813f170 2023-11-24T06:18:57.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.919+0000 7f099ffff700 1 -- 172.21.15.89:0/917902014 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a801f070 con 0x7f09b813f170 2023-11-24T06:18:57.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.919+0000 7f099ffff700 1 -- 172.21.15.89:0/917902014 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a800b8a0 con 0x7f09b813f170 2023-11-24T06:18:57.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.919+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b813f550 con 0x7f09b813f170 2023-11-24T06:18:57.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.920+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b8143910 con 0x7f09b813f170 2023-11-24T06:18:57.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.920+0000 7fef5f7fe700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fef4006af20 0x7fef4006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.920+0000 7f33317fa700 1 -- 172.21.15.89:0/1987188901 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f332c054dc0 con 0x7f33341663d0 2023-11-24T06:18:57.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.920+0000 7f87067fc700 1 -- 172.21.15.89:0/591173846 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f870c02d450 con 0x7f8710162240 2023-11-24T06:18:57.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.920+0000 7f87067fc700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86fc06b4f0 0x7f86fc06d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.920+0000 7f87067fc700 1 -- 172.21.15.89:0/591173846 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f870c08a8a0 con 0x7f8710162240 2023-11-24T06:18:57.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.921+0000 7fef5f7fe700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fef4006af20 0x7fef4006d3e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef601632a0 tx=0x7fef50006c80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.921+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b8067f60 con 0x7f09b813f170 2023-11-24T06:18:57.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.921+0000 7f8715c59700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86fc06b4f0 0x7f86fc06d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.921+0000 7f8715c59700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86fc06b4f0 0x7f86fc06d9b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8700001500 tx=0x7f8700007a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.922+0000 7f099ffff700 1 -- 172.21.15.89:0/917902014 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f09a8021030 con 0x7f09b813f170 2023-11-24T06:18:57.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.922+0000 7f099ffff700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09a406aed0 0x7f09a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.923+0000 7f09b6ffd700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09a406aed0 0x7f09a406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.923+0000 7f099ffff700 1 -- 172.21.15.89:0/917902014 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09a808a4e0 con 0x7f09b813f170 2023-11-24T06:18:57.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.923+0000 7f09b6ffd700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09a406aed0 0x7f09a406d390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0097b0 tx=0x7f09ac006d20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.925+0000 7f87067fc700 1 -- 172.21.15.89:0/591173846 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f870c055ae0 con 0x7f8710162240 2023-11-24T06:18:57.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.926+0000 7f878cff9700 1 -- 172.21.15.89:0/263040466 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8788055d50 con 0x7f879006efc0 2023-11-24T06:18:57.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.927+0000 7f099ffff700 1 -- 172.21.15.89:0/917902014 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09a8055500 con 0x7f09b813f170 2023-11-24T06:18:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.949+0000 7fed5ed69700 1 Processor -- start 2023-11-24T06:18:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.949+0000 7fed5ed69700 1 -- start start 2023-11-24T06:18:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.951+0000 7fed5ed69700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed580b4260 0x7fed5806efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.951+0000 7fed5ed69700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5808bdb0 con 0x7fed58072bb0 2023-11-24T06:18:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.951+0000 7fed5ed69700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5806ce30 con 0x7fed5816ae60 2023-11-24T06:18:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.951+0000 7fed5ed69700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5806d110 con 0x7fed580b4260 2023-11-24T06:18:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.951+0000 7fed5d306700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed580b4260 0x7fed5806efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.951+0000 7fed5d306700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed580b4260 0x7fed5806efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35452/0 (socket says 172.21.15.89:35452) 2023-11-24T06:18:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.951+0000 7fed5d306700 1 -- 172.21.15.89:0/1504290164 learned_addr learned my addr 172.21.15.89:0/1504290164 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.953+0000 7fed5d306700 1 -- 172.21.15.89:0/1504290164 >> v1:172.21.15.123:6789/0 conn(0x7fed5816ae60 legacy=0x7fed580b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.953+0000 7fed5d306700 1 -- 172.21.15.89:0/1504290164 >> v1:172.21.15.89:6789/0 conn(0x7fed58072bb0 legacy=0x7fed58070d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.953+0000 7fed5d306700 1 -- 172.21.15.89:0/1504290164 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5806d3f0 con 0x7fed580b4260 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.954+0000 7fed5d306700 1 --2- 172.21.15.89:0/1504290164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed580b4260 0x7fed5806efc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fed4c006e60 tx=0x7fed4c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a9c552770cfdbe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.953+0000 7fed577fe700 1 -- 172.21.15.89:0/1504290164 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848926944 0 0) 0x7fed5806ce30 con 0x7fed5816ae60 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.954+0000 7fed577fe700 1 -- 172.21.15.89:0/1504290164 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c0092e0 con 0x7fed580b4260 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.954+0000 7fed577fe700 1 -- 172.21.15.89:0/1504290164 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c004030 con 0x7fed580b4260 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.954+0000 7fed577fe700 1 -- 172.21.15.89:0/1504290164 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c0043e0 con 0x7fed580b4260 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.955+0000 7fed5ed69700 1 -- 172.21.15.89:0/1504290164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed580b4260 msgr2=0x7fed5806efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.955+0000 7fed5ed69700 1 --2- 172.21.15.89:0/1504290164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed580b4260 0x7fed5806efc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fed4c006e60 tx=0x7fed4c009b30 comp rx=0 tx=0).stop 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.955+0000 7fed5ed69700 1 -- 172.21.15.89:0/1504290164 shutdown_connections 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.955+0000 7fed5ed69700 1 --2- 172.21.15.89:0/1504290164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed580b4260 0x7fed5806efc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.955+0000 7fed5ed69700 1 -- 172.21.15.89:0/1504290164 >> 172.21.15.89:0/1504290164 conn(0x7fed580cfa30 msgr2=0x7fed580cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.957+0000 7fed5ed69700 1 -- 172.21.15.89:0/1504290164 shutdown_connections 2023-11-24T06:18:57.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.957+0000 7fed5ed69700 1 -- 172.21.15.89:0/1504290164 wait complete. 2023-11-24T06:18:57.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.959+0000 7fed5ed69700 1 Processor -- start 2023-11-24T06:18:57.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.959+0000 7fed5ed69700 1 -- start start 2023-11-24T06:18:57.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.960+0000 7fed5ed69700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed580b4260 0x7fed58207ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.960+0000 7fed5ed69700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed58211600 0x7fed58165f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.960+0000 7fed5ed69700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58211a80 0x7fed58166460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.960+0000 7fed5ed69700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed58212610 con 0x7fed58211600 2023-11-24T06:18:57.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.960+0000 7fed5ed69700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed58212050 con 0x7fed58211a80 2023-11-24T06:18:57.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.960+0000 7fed5ed69700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed58212330 con 0x7fed580b4260 2023-11-24T06:18:57.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.961+0000 7fed5cb05700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed580b4260 0x7fed58207ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.961+0000 7fed5d306700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58211a80 0x7fed58166460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.962+0000 7fed5cb05700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed580b4260 0x7fed58207ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44330/0 (socket says 172.21.15.89:44330) 2023-11-24T06:18:57.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.962+0000 7fed5cb05700 1 -- 172.21.15.89:0/1047942283 learned_addr learned my addr 172.21.15.89:0/1047942283 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:57.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.962+0000 7fed57fff700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed58211600 0x7fed58165f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.963+0000 7fed57fff700 1 -- 172.21.15.89:0/1047942283 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed580b4260 msgr2=0x7fed58207ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.963+0000 7fed57fff700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed580b4260 0x7fed58207ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.963+0000 7fed57fff700 1 -- 172.21.15.89:0/1047942283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58211a80 msgr2=0x7fed58166460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:57.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.963+0000 7fed57fff700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58211a80 0x7fed58166460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:57.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.963+0000 7fed57fff700 1 -- 172.21.15.89:0/1047942283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c00d040 con 0x7fed58211600 2023-11-24T06:18:57.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.963+0000 7fed57fff700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed58211600 0x7fed58165f20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fed5000cbb0 tx=0x7fed5000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.964+0000 7fed557fa700 1 -- 172.21.15.89:0/1047942283 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed50003c00 con 0x7fed58211600 2023-11-24T06:18:57.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.964+0000 7fed557fa700 1 -- 172.21.15.89:0/1047942283 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed50003d60 con 0x7fed58211600 2023-11-24T06:18:57.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.964+0000 7fed557fa700 1 -- 172.21.15.89:0/1047942283 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5000abf0 con 0x7fed58211600 2023-11-24T06:18:57.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.964+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed582128b0 con 0x7fed58211600 2023-11-24T06:18:57.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.964+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed58208410 con 0x7fed58211600 2023-11-24T06:18:57.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.969+0000 7fed557fa700 1 -- 172.21.15.89:0/1047942283 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fed5000ad50 con 0x7fed58211600 2023-11-24T06:18:57.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.970+0000 7fed557fa700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed4406afa0 0x7fed4406d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:57.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.970+0000 7fed557fa700 1 -- 172.21.15.89:0/1047942283 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed50089660 con 0x7fed58211600 2023-11-24T06:18:57.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.970+0000 7fed5cb05700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed4406afa0 0x7fed4406d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:57.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.971+0000 7fed5cb05700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed4406afa0 0x7fed4406d460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fed58214b50 tx=0x7fed48009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:57.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.971+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed380052e0 con 0x7fed58211600 2023-11-24T06:18:57.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:57.981+0000 7fed557fa700 1 -- 172.21.15.89:0/1047942283 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed500546f0 con 0x7fed58211600 2023-11-24T06:18:58.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.249+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2458005590 con 0x7f2464004130 2023-11-24T06:18:58.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.250+0000 7f246b7fe700 1 -- 172.21.15.89:0/1881501886 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f24740582c0 con 0x7f2464004130 2023-11-24T06:18:58.255 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2023-11-24T06:18:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f246406cb80 msgr2=0x7f246406f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f246406cb80 0x7f246406f040 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f246c000c00 tx=0x7f246c013000 comp rx=0 tx=0).stop 2023-11-24T06:18:58.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 msgr2=0x7f247c0aad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f247c0aad70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f247400ccd0 tx=0x7f247400a570 comp rx=0 tx=0).stop 2023-11-24T06:18:58.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 shutdown_connections 2023-11-24T06:18:58.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f246406cb80 0x7f246406f040 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f247c0ce570 0x7f247c15b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2464004130 0x7f247c0aad70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 --2- 172.21.15.89:0/1881501886 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f247c0ce0f0 0x7f247c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.254+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 >> 172.21.15.89:0/1881501886 conn(0x7f247c0cfa30 msgr2=0x7f247c1402e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.255+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 shutdown_connections 2023-11-24T06:18:58.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.255+0000 7f24817f2700 1 -- 172.21.15.89:0/1881501886 wait complete. 2023-11-24T06:18:58.275 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.3 2023-11-24T06:18:58.275 DEBUG:teuthology.parallel:result is None 2023-11-24T06:18:58.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.308+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6f30005590 con 0x7f6f48071be0 2023-11-24T06:18:58.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.308+0000 7f6f3e7fc700 1 -- 172.21.15.89:0/694454641 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6f40059ae0 con 0x7f6f48071be0 2023-11-24T06:18:58.311 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2023-11-24T06:18:58.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6f2c06afa0 msgr2=0x7f6f2c06d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6f2c06afa0 0x7f6f2c06d460 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f440010e0 tx=0x7f6f440079e0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f48071be0 msgr2=0x7f6f481fbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f48071be0 0x7f6f481fbe20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6f40017000 tx=0x7f6f40004ad0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 shutdown_connections 2023-11-24T06:18:58.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6f2c06afa0 0x7f6f2c06d460 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f4815d6a0 0x7f6f480aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f48071be0 0x7f6f481fbe20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 --2- 172.21.15.89:0/694454641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f480b4090 0x7f6f481f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.312+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 >> 172.21.15.89:0/694454641 conn(0x7f6f480cfa30 msgr2=0x7f6f48140fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.314+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 shutdown_connections 2023-11-24T06:18:58.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.314+0000 7f6f4fc14700 1 -- 172.21.15.89:0/694454641 wait complete. 2023-11-24T06:18:58.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.316+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fef600ce550 con 0x7fef6006d4d0 2023-11-24T06:18:58.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.321+0000 7fef5cff9700 1 -- 172.21.15.89:0/3629446841 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fef600ce550 con 0x7fef6006d4d0 2023-11-24T06:18:58.325 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2023-11-24T06:18:58.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.324+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f33340751e0 con 0x7f33341663d0 2023-11-24T06:18:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.326+0000 7f33317fa700 1 -- 172.21.15.89:0/1987188901 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f332c023090 con 0x7f33341663d0 2023-11-24T06:18:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.326+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fef4006af20 msgr2=0x7fef4006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.326+0000 7fef66357700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fef4006af20 0x7fef4006d3e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef601632a0 tx=0x7fef50006c80 comp rx=0 tx=0).stop 2023-11-24T06:18:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.326+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef6006d4d0 msgr2=0x7fef601388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.326+0000 7fef66357700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef6006d4d0 0x7fef601388d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fef5800c8c0 tx=0x7fef5800cbd0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.330 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2023-11-24T06:18:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.327+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 shutdown_connections 2023-11-24T06:18:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.327+0000 7fef66357700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef60164a90 0x7fef6014a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.327+0000 7fef66357700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fef4006af20 0x7fef4006d3e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.327+0000 7fef66357700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef6006d4d0 0x7fef601388d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.327+0000 7fef66357700 1 --2- 172.21.15.89:0/3629446841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef600b3b50 0x7fef600cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.327+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 >> 172.21.15.89:0/3629446841 conn(0x7fef600cfa30 msgr2=0x7fef600b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.328+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 shutdown_connections 2023-11-24T06:18:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.328+0000 7fef66357700 1 -- 172.21.15.89:0/3629446841 wait complete. 2023-11-24T06:18:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.331+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f86f80051a0 con 0x7f8710162240 2023-11-24T06:18:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.331+0000 7f87067fc700 1 -- 172.21.15.89:0/591173846 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f870c015020 con 0x7f8710162240 2023-11-24T06:18:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.332+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f332006af20 msgr2=0x7f332006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.332+0000 7f333aaff700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f332006af20 0x7f332006d3e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3334145910 tx=0x7f3324009210 comp rx=0 tx=0).stop 2023-11-24T06:18:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.332+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33341663d0 msgr2=0x7f333414d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.332+0000 7f333aaff700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33341663d0 0x7f333414d060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f332c00be20 tx=0x7f332c009ef0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.6 2023-11-24T06:18:58.335 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2023-11-24T06:18:58.335 DEBUG:teuthology.parallel:result is None 2023-11-24T06:18:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 shutdown_connections 2023-11-24T06:18:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f333aaff700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f332006af20 0x7f332006d3e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f333aaff700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33341663d0 0x7f333414d060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f333aaff700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334070fd0 0x7f333417ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f333aaff700 1 --2- 172.21.15.89:0/1987188901 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf440 0x7f33340b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 >> 172.21.15.89:0/1987188901 conn(0x7f33340b1740 msgr2=0x7f333414e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 shutdown_connections 2023-11-24T06:18:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f333aaff700 1 -- 172.21.15.89:0/1987188901 wait complete. 2023-11-24T06:18:58.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.334+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f09b8067f60 con 0x7f09b813f170 2023-11-24T06:18:58.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.335+0000 7f099ffff700 1 -- 172.21.15.89:0/917902014 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f09a8058b20 con 0x7f09b813f170 2023-11-24T06:18:58.337 INFO:teuthology.orchestra.run.smithi089.stdout:55834574857 2023-11-24T06:18:58.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.335+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f87900751e0 con 0x7f879006efc0 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.335+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86fc06b4f0 msgr2=0x7f86fc06d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86fc06b4f0 0x7f86fc06d9b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8700001500 tx=0x7f8700007a20 comp rx=0 tx=0).stop 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8710162240 msgr2=0x7f87101634a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8710162240 0x7f87101634a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f870c0090b0 tx=0x7f870c004750 comp rx=0 tx=0).stop 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 shutdown_connections 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86fc06b4f0 0x7f86fc06d9b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87101593f0 0x7f871014b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871006f360 0x7f871015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 --2- 172.21.15.89:0/591173846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8710162240 0x7f87101634a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 >> 172.21.15.89:0/591173846 conn(0x7f8710071b90 msgr2=0x7f87100b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f878cff9700 1 -- 172.21.15.89:0/263040466 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8788021020 con 0x7f879006efc0 2023-11-24T06:18:58.340 INFO:teuthology.orchestra.run.smithi089.stdout:55834574857 2023-11-24T06:18:58.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 shutdown_connections 2023-11-24T06:18:58.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.336+0000 7f8717ebd700 1 -- 172.21.15.89:0/591173846 wait complete. 2023-11-24T06:18:58.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 msgr2=0x7f877006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8790148000 tx=0x7f8780009190 comp rx=0 tx=0).stop 2023-11-24T06:18:58.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879006efc0 msgr2=0x7f8790161d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879006efc0 0x7f8790161d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f879014c330 tx=0x7f8788004ad0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09a406aed0 msgr2=0x7f09a406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f09bd9af700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09a406aed0 0x7f09a406d390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0097b0 tx=0x7f09ac006d20 comp rx=0 tx=0).stop 2023-11-24T06:18:58.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b813f170 msgr2=0x7f09b8143d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f09bd9af700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b813f170 0x7f09b8143d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09a8015000 tx=0x7f09a8004750 comp rx=0 tx=0).stop 2023-11-24T06:18:58.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 shutdown_connections 2023-11-24T06:18:58.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879014be10 0x7f87901fbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879006efc0 0x7f8790161d90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 --2- 172.21.15.89:0/263040466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900b4260 0x7f8790143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 >> 172.21.15.89:0/263040466 conn(0x7f87900cfa30 msgr2=0x7f8790148f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 shutdown_connections 2023-11-24T06:18:58.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.339+0000 7f8796979700 1 -- 172.21.15.89:0/263040466 wait complete. 2023-11-24T06:18:58.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.340+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 shutdown_connections 2023-11-24T06:18:58.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.340+0000 7f09bd9af700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b8160440 0x7f09b81442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.340+0000 7f09bd9af700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b806c540 0x7f09b8152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.340+0000 7f09bd9af700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09a406aed0 0x7f09a406d390 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.340+0000 7f09bd9af700 1 --2- 172.21.15.89:0/917902014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b813f170 0x7f09b8143d80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.340+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 >> 172.21.15.89:0/917902014 conn(0x7f09b8071ac0 msgr2=0x7f09b80b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.342+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fed38005590 con 0x7fed58211600 2023-11-24T06:18:58.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.342+0000 7fed557fa700 1 -- 172.21.15.89:0/1047942283 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fed50057d10 con 0x7fed58211600 2023-11-24T06:18:58.345 INFO:teuthology.orchestra.run.smithi089.stdout:55834574857 2023-11-24T06:18:58.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.343+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 shutdown_connections 2023-11-24T06:18:58.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.343+0000 7f09bd9af700 1 -- 172.21.15.89:0/917902014 wait complete. 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed4406afa0 msgr2=0x7fed4406d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed4406afa0 0x7fed4406d460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fed58214b50 tx=0x7fed48009040 comp rx=0 tx=0).stop 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed58211600 msgr2=0x7fed58165f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed58211600 0x7fed58165f20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fed5000cbb0 tx=0x7fed5000cf70 comp rx=0 tx=0).stop 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 shutdown_connections 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed4406afa0 0x7fed4406d460 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58211a80 0x7fed58166460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed580b4260 0x7fed58207ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 --2- 172.21.15.89:0/1047942283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed58211600 0x7fed58165f20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 >> 172.21.15.89:0/1047942283 conn(0x7fed580cfa30 msgr2=0x7fed58162c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 shutdown_connections 2023-11-24T06:18:58.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.345+0000 7fed5ed69700 1 -- 172.21.15.89:0/1047942283 wait complete. 2023-11-24T06:18:58.349 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.5 2023-11-24T06:18:58.350 DEBUG:teuthology.parallel:result is None 2023-11-24T06:18:58.359 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.7 2023-11-24T06:18:58.359 DEBUG:teuthology.parallel:result is None 2023-11-24T06:18:58.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.2 2023-11-24T06:18:58.363 DEBUG:teuthology.parallel:result is None 2023-11-24T06:18:58.364 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.0 2023-11-24T06:18:58.364 DEBUG:teuthology.parallel:result is None 2023-11-24T06:18:58.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.1 2023-11-24T06:18:58.365 DEBUG:teuthology.parallel:result is None 2023-11-24T06:18:58.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.4 2023-11-24T06:18:58.366 DEBUG:teuthology.parallel:result is None 2023-11-24T06:18:58.366 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-24T06:18:58.366 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:18:58.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.522+0000 7efff8a83700 1 Processor -- start 2023-11-24T06:18:58.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.522+0000 7efff8a83700 1 -- start start 2023-11-24T06:18:58.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.523+0000 7efff8a83700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efff415e080 0x7efff40cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.523+0000 7efff8a83700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff40d0150 con 0x7efff40ce4d0 2023-11-24T06:18:58.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.523+0000 7efff8a83700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff40cee50 con 0x7efff40d22f0 2023-11-24T06:18:58.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.523+0000 7efff8a83700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff415e460 con 0x7efff415e080 2023-11-24T06:18:58.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.523+0000 7efff259c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efff415e080 0x7efff40cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.523+0000 7efff259c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efff415e080 0x7efff40cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35462/0 (socket says 172.21.15.89:35462) 2023-11-24T06:18:58.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.523+0000 7efff259c700 1 -- 172.21.15.89:0/3921308818 learned_addr learned my addr 172.21.15.89:0/3921308818 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:58.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.523+0000 7efff1d9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efff40ce4d0 0x7efff40d1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38968/0 (socket says 172.21.15.89:38968) 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff259c700 1 -- 172.21.15.89:0/3921308818 >> v1:172.21.15.123:6789/0 conn(0x7efff40d22f0 legacy=0x7efff40d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff259c700 1 -- 172.21.15.89:0/3921308818 >> v1:172.21.15.89:6789/0 conn(0x7efff40ce4d0 legacy=0x7efff40d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff259c700 1 -- 172.21.15.89:0/3921308818 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff40cf130 con 0x7efff415e080 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff159a700 1 -- 172.21.15.89:0/3921308818 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404967014 0 0) 0x7efff40d0150 con 0x7efff40ce4d0 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff259c700 1 --2- 172.21.15.89:0/3921308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40cc2a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7effe80098f0 tx=0x7effe8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd2fc6d2a2a690b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff159a700 1 -- 172.21.15.89:0/3921308818 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe800b8a0 con 0x7efff415e080 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff159a700 1 -- 172.21.15.89:0/3921308818 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe800ba00 con 0x7efff415e080 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff159a700 1 -- 172.21.15.89:0/3921308818 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe800bd90 con 0x7efff415e080 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff8a83700 1 -- 172.21.15.89:0/3921308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 msgr2=0x7efff40cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.524+0000 7efff8a83700 1 --2- 172.21.15.89:0/3921308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40cc2a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7effe80098f0 tx=0x7effe8009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:58.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.525+0000 7efff8a83700 1 -- 172.21.15.89:0/3921308818 shutdown_connections 2023-11-24T06:18:58.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.525+0000 7efff8a83700 1 --2- 172.21.15.89:0/3921308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40cc2a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.525+0000 7efff8a83700 1 -- 172.21.15.89:0/3921308818 >> 172.21.15.89:0/3921308818 conn(0x7efff4073130 msgr2=0x7efff409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.525+0000 7efff8a83700 1 -- 172.21.15.89:0/3921308818 shutdown_connections 2023-11-24T06:18:58.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.525+0000 7efff8a83700 1 -- 172.21.15.89:0/3921308818 wait complete. 2023-11-24T06:18:58.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.526+0000 7efff8a83700 1 Processor -- start 2023-11-24T06:18:58.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.526+0000 7efff8a83700 1 -- start start 2023-11-24T06:18:58.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff8a83700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efff40ce4d0 0x7efff41501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff8a83700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efff40d22f0 0x7efff40c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff8a83700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff8a83700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff40c64e0 con 0x7efff415e080 2023-11-24T06:18:58.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff8a83700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff40c67c0 con 0x7efff40ce4d0 2023-11-24T06:18:58.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff8a83700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff40c6aa0 con 0x7efff40d22f0 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35472/0 (socket says 172.21.15.89:35472) 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 -- 172.21.15.89:0/2706165217 learned_addr learned my addr 172.21.15.89:0/2706165217 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff259c700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efff40ce4d0 0x7efff41501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff1d9b700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efff40d22f0 0x7efff40c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 -- 172.21.15.89:0/2706165217 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efff40d22f0 msgr2=0x7efff40c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efff40d22f0 0x7efff40c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 -- 172.21.15.89:0/2706165217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efff40ce4d0 msgr2=0x7efff41501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efff40ce4d0 0x7efff41501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 -- 172.21.15.89:0/2706165217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe8009550 con 0x7efff415e080 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff2d9d700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40caa30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7effec00cc50 tx=0x7effec00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7effe2ffd700 1 -- 172.21.15.89:0/2706165217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec00ad80 con 0x7efff415e080 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7effe2ffd700 1 -- 172.21.15.89:0/2706165217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effec00ce50 con 0x7efff415e080 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff40c6d80 con 0x7efff415e080 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7effe2ffd700 1 -- 172.21.15.89:0/2706165217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec003c40 con 0x7efff415e080 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.527+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff40c6120 con 0x7efff415e080 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.528+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff4067f60 con 0x7efff415e080 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.529+0000 7effe2ffd700 1 -- 172.21.15.89:0/2706165217 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7effec0050d0 con 0x7efff415e080 2023-11-24T06:18:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.529+0000 7effe2ffd700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7effd406aed0 0x7effd406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.529+0000 7effe2ffd700 1 -- 172.21.15.89:0/2706165217 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effec089670 con 0x7efff415e080 2023-11-24T06:18:58.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.529+0000 7efff259c700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7effd406aed0 0x7effd406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.529+0000 7efff259c700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7effd406aed0 0x7effd406d390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7effe8009130 tx=0x7effe801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:58.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.534+0000 7effe2ffd700 1 -- 172.21.15.89:0/2706165217 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effec0547b0 con 0x7efff415e080 2023-11-24T06:18:58.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.720+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efff40c6930 con 0x7effd406aed0 2023-11-24T06:18:58.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.727+0000 7effe2ffd700 1 -- 172.21.15.89:0/2706165217 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264578 (secure 0 0 0) 0x7efff40c6930 con 0x7effd406aed0 2023-11-24T06:18:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:58.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7effd406aed0 msgr2=0x7effd406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7effd406aed0 0x7effd406d390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7effe8009130 tx=0x7effe801c040 comp rx=0 tx=0).stop 2023-11-24T06:18:58.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 msgr2=0x7efff40caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40caa30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7effec00cc50 tx=0x7effec00a340 comp rx=0 tx=0).stop 2023-11-24T06:18:58.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 shutdown_connections 2023-11-24T06:18:58.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff415e080 0x7efff40caa30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7effd406aed0 0x7effd406d390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efff40ce4d0 0x7efff41501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 --2- 172.21.15.89:0/2706165217 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efff40d22f0 0x7efff40c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 >> 172.21.15.89:0/2706165217 conn(0x7efff4073130 msgr2=0x7efff409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 shutdown_connections 2023-11-24T06:18:58.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.730+0000 7efff8a83700 1 -- 172.21.15.89:0/2706165217 wait complete. 2023-11-24T06:18:58.735 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:18:58.744 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-24T06:18:57.721114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219104,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755616,"statfs":{"total":773094113280,"available":772869750784,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003267"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024214+0000","last_change":"2023-11-24T06:18:50.033723+0000","last_active":"2023-11-24T06:18:51.024214+0000","last_peered":"2023-11-24T06:18:51.024214+0000","last_clean":"2023-11-24T06:18:51.024214+0000","last_became_active":"2023-11-24T06:18:50.033433+0000","last_became_peered":"2023-11-24T06:18:50.033433+0000","last_unstale":"2023-11-24T06:18:51.024214+0000","last_undegraded":"2023-11-24T06:18:51.024214+0000","last_fullsized":"2023-11-24T06:18:51.024214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:49.822079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021557+0000","last_change":"2023-11-24T06:18:50.024643+0000","last_active":"2023-11-24T06:18:51.021557+0000","last_peered":"2023-11-24T06:18:51.021557+0000","last_clean":"2023-11-24T06:18:51.021557+0000","last_became_active":"2023-11-24T06:18:50.024352+0000","last_became_peered":"2023-11-24T06:18:50.024352+0000","last_unstale":"2023-11-24T06:18:51.021557+0000","last_undegraded":"2023-11-24T06:18:51.021557+0000","last_fullsized":"2023-11-24T06:18:51.021557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:59.327185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403187+0000","last_change":"2023-11-24T06:18:50.025637+0000","last_active":"2023-11-24T06:18:52.403187+0000","last_peered":"2023-11-24T06:18:52.403187+0000","last_clean":"2023-11-24T06:18:52.403187+0000","last_became_active":"2023-11-24T06:18:50.025328+0000","last_became_peered":"2023-11-24T06:18:50.025328+0000","last_unstale":"2023-11-24T06:18:52.403187+0000","last_undegraded":"2023-11-24T06:18:52.403187+0000","last_fullsized":"2023-11-24T06:18:52.403187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:41.341947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892759+0000","last_change":"2023-11-24T06:18:50.030667+0000","last_active":"2023-11-24T06:18:51.892759+0000","last_peered":"2023-11-24T06:18:51.892759+0000","last_clean":"2023-11-24T06:18:51.892759+0000","last_became_active":"2023-11-24T06:18:50.030133+0000","last_became_peered":"2023-11-24T06:18:50.030133+0000","last_unstale":"2023-11-24T06:18:51.892759+0000","last_undegraded":"2023-11-24T06:18:51.892759+0000","last_fullsized":"2023-11-24T06:18:51.892759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:21.299524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024116+0000","last_change":"2023-11-24T06:18:50.025413+0000","last_active":"2023-11-24T06:18:51.024116+0000","last_peered":"2023-11-24T06:18:51.024116+0000","last_clean":"2023-11-24T06:18:51.024116+0000","last_became_active":"2023-11-24T06:18:50.025118+0000","last_became_peered":"2023-11-24T06:18:50.025118+0000","last_unstale":"2023-11-24T06:18:51.024116+0000","last_undegraded":"2023-11-24T06:18:51.024116+0000","last_fullsized":"2023-11-24T06:18:51.024116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:21.299524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892331+0000","last_change":"2023-11-24T06:18:50.039892+0000","last_active":"2023-11-24T06:18:51.892331+0000","last_peered":"2023-11-24T06:18:51.892331+0000","last_clean":"2023-11-24T06:18:51.892331+0000","last_became_active":"2023-11-24T06:18:50.039525+0000","last_became_peered":"2023-11-24T06:18:50.039525+0000","last_unstale":"2023-11-24T06:18:51.892331+0000","last_undegraded":"2023-11-24T06:18:51.892331+0000","last_fullsized":"2023-11-24T06:18:51.892331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022843+0000","last_change":"2023-11-24T06:18:50.024213+0000","last_active":"2023-11-24T06:18:51.022843+0000","last_peered":"2023-11-24T06:18:51.022843+0000","last_clean":"2023-11-24T06:18:51.022843+0000","last_became_active":"2023-11-24T06:18:50.023914+0000","last_became_peered":"2023-11-24T06:18:50.023914+0000","last_unstale":"2023-11-24T06:18:51.022843+0000","last_undegraded":"2023-11-24T06:18:51.022843+0000","last_fullsized":"2023-11-24T06:18:51.022843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:48.527966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.402897+0000","last_change":"2023-11-24T06:18:50.037907+0000","last_active":"2023-11-24T06:18:52.402897+0000","last_peered":"2023-11-24T06:18:52.402897+0000","last_clean":"2023-11-24T06:18:52.402897+0000","last_became_active":"2023-11-24T06:18:50.037606+0000","last_became_peered":"2023-11-24T06:18:50.037606+0000","last_unstale":"2023-11-24T06:18:52.402897+0000","last_undegraded":"2023-11-24T06:18:52.402897+0000","last_fullsized":"2023-11-24T06:18:52.402897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:55.342818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024189+0000","last_change":"2023-11-24T06:18:50.035382+0000","last_active":"2023-11-24T06:18:51.024189+0000","last_peered":"2023-11-24T06:18:51.024189+0000","last_clean":"2023-11-24T06:18:51.024189+0000","last_became_active":"2023-11-24T06:18:50.035148+0000","last_became_peered":"2023-11-24T06:18:50.035148+0000","last_unstale":"2023-11-24T06:18:51.024189+0000","last_undegraded":"2023-11-24T06:18:51.024189+0000","last_fullsized":"2023-11-24T06:18:51.024189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:39.141407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328435+0000","last_change":"2023-11-24T06:18:50.037425+0000","last_active":"2023-11-24T06:18:52.328435+0000","last_peered":"2023-11-24T06:18:52.328435+0000","last_clean":"2023-11-24T06:18:52.328435+0000","last_became_active":"2023-11-24T06:18:50.037111+0000","last_became_peered":"2023-11-24T06:18:50.037111+0000","last_unstale":"2023-11-24T06:18:52.328435+0000","last_undegraded":"2023-11-24T06:18:52.328435+0000","last_fullsized":"2023-11-24T06:18:52.328435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328117+0000","last_change":"2023-11-24T06:18:50.036955+0000","last_active":"2023-11-24T06:18:52.328117+0000","last_peered":"2023-11-24T06:18:52.328117+0000","last_clean":"2023-11-24T06:18:52.328117+0000","last_became_active":"2023-11-24T06:18:50.036643+0000","last_became_peered":"2023-11-24T06:18:50.036643+0000","last_unstale":"2023-11-24T06:18:52.328117+0000","last_undegraded":"2023-11-24T06:18:52.328117+0000","last_fullsized":"2023-11-24T06:18:52.328117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023821+0000","last_change":"2023-11-24T06:18:50.034218+0000","last_active":"2023-11-24T06:18:51.023821+0000","last_peered":"2023-11-24T06:18:51.023821+0000","last_clean":"2023-11-24T06:18:51.023821+0000","last_became_active":"2023-11-24T06:18:50.033754+0000","last_became_peered":"2023-11-24T06:18:50.033754+0000","last_unstale":"2023-11-24T06:18:51.023821+0000","last_undegraded":"2023-11-24T06:18:51.023821+0000","last_fullsized":"2023-11-24T06:18:51.023821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403228+0000","last_change":"2023-11-24T06:18:50.033709+0000","last_active":"2023-11-24T06:18:52.403228+0000","last_peered":"2023-11-24T06:18:52.403228+0000","last_clean":"2023-11-24T06:18:52.403228+0000","last_became_active":"2023-11-24T06:18:50.033447+0000","last_became_peered":"2023-11-24T06:18:50.033447+0000","last_unstale":"2023-11-24T06:18:52.403228+0000","last_undegraded":"2023-11-24T06:18:52.403228+0000","last_fullsized":"2023-11-24T06:18:52.403228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403761+0000","last_change":"2023-11-24T06:18:50.023134+0000","last_active":"2023-11-24T06:18:52.403761+0000","last_peered":"2023-11-24T06:18:52.403761+0000","last_clean":"2023-11-24T06:18:52.403761+0000","last_became_active":"2023-11-24T06:18:50.022837+0000","last_became_peered":"2023-11-24T06:18:50.022837+0000","last_unstale":"2023-11-24T06:18:52.403761+0000","last_undegraded":"2023-11-24T06:18:52.403761+0000","last_fullsized":"2023-11-24T06:18:52.403761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021102+0000","last_change":"2023-11-24T06:18:50.027089+0000","last_active":"2023-11-24T06:18:51.021102+0000","last_peered":"2023-11-24T06:18:51.021102+0000","last_clean":"2023-11-24T06:18:51.021102+0000","last_became_active":"2023-11-24T06:18:50.026810+0000","last_became_peered":"2023-11-24T06:18:50.026810+0000","last_unstale":"2023-11-24T06:18:51.021102+0000","last_undegraded":"2023-11-24T06:18:51.021102+0000","last_fullsized":"2023-11-24T06:18:51.021102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892218+0000","last_change":"2023-11-24T06:18:50.028542+0000","last_active":"2023-11-24T06:18:51.892218+0000","last_peered":"2023-11-24T06:18:51.892218+0000","last_clean":"2023-11-24T06:18:51.892218+0000","last_became_active":"2023-11-24T06:18:50.027971+0000","last_became_peered":"2023-11-24T06:18:50.027971+0000","last_unstale":"2023-11-24T06:18:51.892218+0000","last_undegraded":"2023-11-24T06:18:51.892218+0000","last_fullsized":"2023-11-24T06:18:51.892218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892968+0000","last_change":"2023-11-24T06:18:50.035313+0000","last_active":"2023-11-24T06:18:51.892968+0000","last_peered":"2023-11-24T06:18:51.892968+0000","last_clean":"2023-11-24T06:18:51.892968+0000","last_became_active":"2023-11-24T06:18:50.034744+0000","last_became_peered":"2023-11-24T06:18:50.034744+0000","last_unstale":"2023-11-24T06:18:51.892968+0000","last_undegraded":"2023-11-24T06:18:51.892968+0000","last_fullsized":"2023-11-24T06:18:51.892968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:03.577619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403701+0000","last_change":"2023-11-24T06:18:50.034072+0000","last_active":"2023-11-24T06:18:52.403701+0000","last_peered":"2023-11-24T06:18:52.403701+0000","last_clean":"2023-11-24T06:18:52.403701+0000","last_became_active":"2023-11-24T06:18:50.033798+0000","last_became_peered":"2023-11-24T06:18:50.033798+0000","last_unstale":"2023-11-24T06:18:52.403701+0000","last_undegraded":"2023-11-24T06:18:52.403701+0000","last_fullsized":"2023-11-24T06:18:52.403701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023796+0000","last_change":"2023-11-24T06:18:50.025458+0000","last_active":"2023-11-24T06:18:51.023796+0000","last_peered":"2023-11-24T06:18:51.023796+0000","last_clean":"2023-11-24T06:18:51.023796+0000","last_became_active":"2023-11-24T06:18:50.025139+0000","last_became_peered":"2023-11-24T06:18:50.025139+0000","last_unstale":"2023-11-24T06:18:51.023796+0000","last_undegraded":"2023-11-24T06:18:51.023796+0000","last_fullsized":"2023-11-24T06:18:51.023796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:02:29.652618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.735806+0000","last_change":"2023-11-24T06:18:49.009105+0000","last_active":"2023-11-24T06:18:51.735806+0000","last_peered":"2023-11-24T06:18:51.735806+0000","last_clean":"2023-11-24T06:18:51.735806+0000","last_became_active":"2023-11-24T06:18:49.008802+0000","last_became_peered":"2023-11-24T06:18:49.008802+0000","last_unstale":"2023-11-24T06:18:51.735806+0000","last_undegraded":"2023-11-24T06:18:51.735806+0000","last_fullsized":"2023-11-24T06:18:51.735806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403494+0000","last_change":"2023-11-24T06:18:50.033700+0000","last_active":"2023-11-24T06:18:52.403494+0000","last_peered":"2023-11-24T06:18:52.403494+0000","last_clean":"2023-11-24T06:18:52.403494+0000","last_became_active":"2023-11-24T06:18:50.033372+0000","last_became_peered":"2023-11-24T06:18:50.033372+0000","last_unstale":"2023-11-24T06:18:52.403494+0000","last_undegraded":"2023-11-24T06:18:52.403494+0000","last_fullsized":"2023-11-24T06:18:52.403494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:14.887594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.739276+0000","last_change":"2023-11-24T06:18:49.015649+0000","last_active":"2023-11-24T06:18:51.739276+0000","last_peered":"2023-11-24T06:18:51.739276+0000","last_clean":"2023-11-24T06:18:51.739276+0000","last_became_active":"2023-11-24T06:18:49.015329+0000","last_became_peered":"2023-11-24T06:18:49.015329+0000","last_unstale":"2023-11-24T06:18:51.739276+0000","last_undegraded":"2023-11-24T06:18:51.739276+0000","last_fullsized":"2023-11-24T06:18:51.739276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403874+0000","last_change":"2023-11-24T06:18:50.037859+0000","last_active":"2023-11-24T06:18:52.403874+0000","last_peered":"2023-11-24T06:18:52.403874+0000","last_clean":"2023-11-24T06:18:52.403874+0000","last_became_active":"2023-11-24T06:18:50.037565+0000","last_became_peered":"2023-11-24T06:18:50.037565+0000","last_unstale":"2023-11-24T06:18:52.403874+0000","last_undegraded":"2023-11-24T06:18:52.403874+0000","last_fullsized":"2023-11-24T06:18:52.403874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024207+0000","last_change":"2023-11-24T06:18:49.008317+0000","last_active":"2023-11-24T06:18:51.024207+0000","last_peered":"2023-11-24T06:18:51.024207+0000","last_clean":"2023-11-24T06:18:51.024207+0000","last_became_active":"2023-11-24T06:18:49.008039+0000","last_became_peered":"2023-11-24T06:18:49.008039+0000","last_unstale":"2023-11-24T06:18:51.024207+0000","last_undegraded":"2023-11-24T06:18:51.024207+0000","last_fullsized":"2023-11-24T06:18:51.024207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:58.320250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892514+0000","last_change":"2023-11-24T06:18:50.034882+0000","last_active":"2023-11-24T06:18:51.892514+0000","last_peered":"2023-11-24T06:18:51.892514+0000","last_clean":"2023-11-24T06:18:51.892514+0000","last_became_active":"2023-11-24T06:18:50.031623+0000","last_became_peered":"2023-11-24T06:18:50.031623+0000","last_unstale":"2023-11-24T06:18:51.892514+0000","last_undegraded":"2023-11-24T06:18:51.892514+0000","last_fullsized":"2023-11-24T06:18:51.892514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:14.887594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023720+0000","last_change":"2023-11-24T06:18:49.007346+0000","last_active":"2023-11-24T06:18:51.023720+0000","last_peered":"2023-11-24T06:18:51.023720+0000","last_clean":"2023-11-24T06:18:51.023720+0000","last_became_active":"2023-11-24T06:18:49.006300+0000","last_became_peered":"2023-11-24T06:18:49.006300+0000","last_unstale":"2023-11-24T06:18:51.023720+0000","last_undegraded":"2023-11-24T06:18:51.023720+0000","last_fullsized":"2023-11-24T06:18:51.023720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892742+0000","last_change":"2023-11-24T06:18:50.029883+0000","last_active":"2023-11-24T06:18:51.892742+0000","last_peered":"2023-11-24T06:18:51.892742+0000","last_clean":"2023-11-24T06:18:51.892742+0000","last_became_active":"2023-11-24T06:18:50.029618+0000","last_became_peered":"2023-11-24T06:18:50.029618+0000","last_unstale":"2023-11-24T06:18:51.892742+0000","last_undegraded":"2023-11-24T06:18:51.892742+0000","last_fullsized":"2023-11-24T06:18:51.892742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:32:20.935789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.739242+0000","last_change":"2023-11-24T06:18:49.015809+0000","last_active":"2023-11-24T06:18:51.739242+0000","last_peered":"2023-11-24T06:18:51.739242+0000","last_clean":"2023-11-24T06:18:51.739242+0000","last_became_active":"2023-11-24T06:18:49.015073+0000","last_became_peered":"2023-11-24T06:18:49.015073+0000","last_unstale":"2023-11-24T06:18:51.739242+0000","last_undegraded":"2023-11-24T06:18:51.739242+0000","last_fullsized":"2023-11-24T06:18:51.739242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:40.335012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023148+0000","last_change":"2023-11-24T06:18:50.015301+0000","last_active":"2023-11-24T06:18:51.023148+0000","last_peered":"2023-11-24T06:18:51.023148+0000","last_clean":"2023-11-24T06:18:51.023148+0000","last_became_active":"2023-11-24T06:18:50.015007+0000","last_became_peered":"2023-11-24T06:18:50.015007+0000","last_unstale":"2023-11-24T06:18:51.023148+0000","last_undegraded":"2023-11-24T06:18:51.023148+0000","last_fullsized":"2023-11-24T06:18:51.023148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738456+0000","last_change":"2023-11-24T06:18:49.008047+0000","last_active":"2023-11-24T06:18:51.738456+0000","last_peered":"2023-11-24T06:18:51.738456+0000","last_clean":"2023-11-24T06:18:51.738456+0000","last_became_active":"2023-11-24T06:18:49.007764+0000","last_became_peered":"2023-11-24T06:18:49.007764+0000","last_unstale":"2023-11-24T06:18:51.738456+0000","last_undegraded":"2023-11-24T06:18:51.738456+0000","last_fullsized":"2023-11-24T06:18:51.738456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:02.607697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403873+0000","last_change":"2023-11-24T06:18:50.027945+0000","last_active":"2023-11-24T06:18:52.403873+0000","last_peered":"2023-11-24T06:18:52.403873+0000","last_clean":"2023-11-24T06:18:52.403873+0000","last_became_active":"2023-11-24T06:18:50.027609+0000","last_became_peered":"2023-11-24T06:18:50.027609+0000","last_unstale":"2023-11-24T06:18:52.403873+0000","last_undegraded":"2023-11-24T06:18:52.403873+0000","last_fullsized":"2023-11-24T06:18:52.403873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.739190+0000","last_change":"2023-11-24T06:18:49.014611+0000","last_active":"2023-11-24T06:18:51.739190+0000","last_peered":"2023-11-24T06:18:51.739190+0000","last_clean":"2023-11-24T06:18:51.739190+0000","last_became_active":"2023-11-24T06:18:49.014277+0000","last_became_peered":"2023-11-24T06:18:49.014277+0000","last_unstale":"2023-11-24T06:18:51.739190+0000","last_undegraded":"2023-11-24T06:18:51.739190+0000","last_fullsized":"2023-11-24T06:18:51.739190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:33.478815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.402995+0000","last_change":"2023-11-24T06:18:50.027283+0000","last_active":"2023-11-24T06:18:52.402995+0000","last_peered":"2023-11-24T06:18:52.402995+0000","last_clean":"2023-11-24T06:18:52.402995+0000","last_became_active":"2023-11-24T06:18:50.027028+0000","last_became_peered":"2023-11-24T06:18:50.027028+0000","last_unstale":"2023-11-24T06:18:52.402995+0000","last_undegraded":"2023-11-24T06:18:52.402995+0000","last_fullsized":"2023-11-24T06:18:52.402995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738445+0000","last_change":"2023-11-24T06:18:49.007327+0000","last_active":"2023-11-24T06:18:51.738445+0000","last_peered":"2023-11-24T06:18:51.738445+0000","last_clean":"2023-11-24T06:18:51.738445+0000","last_became_active":"2023-11-24T06:18:49.006684+0000","last_became_peered":"2023-11-24T06:18:49.006684+0000","last_unstale":"2023-11-24T06:18:51.738445+0000","last_undegraded":"2023-11-24T06:18:51.738445+0000","last_fullsized":"2023-11-24T06:18:51.738445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:54.335883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021189+0000","last_change":"2023-11-24T06:18:50.026294+0000","last_active":"2023-11-24T06:18:51.021189+0000","last_peered":"2023-11-24T06:18:51.021189+0000","last_clean":"2023-11-24T06:18:51.021189+0000","last_became_active":"2023-11-24T06:18:50.026020+0000","last_became_peered":"2023-11-24T06:18:50.026020+0000","last_unstale":"2023-11-24T06:18:51.021189+0000","last_undegraded":"2023-11-24T06:18:51.021189+0000","last_fullsized":"2023-11-24T06:18:51.021189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738579+0000","last_change":"2023-11-24T06:18:49.011149+0000","last_active":"2023-11-24T06:18:51.738579+0000","last_peered":"2023-11-24T06:18:51.738579+0000","last_clean":"2023-11-24T06:18:51.738579+0000","last_became_active":"2023-11-24T06:18:49.010573+0000","last_became_peered":"2023-11-24T06:18:49.010573+0000","last_unstale":"2023-11-24T06:18:51.738579+0000","last_undegraded":"2023-11-24T06:18:51.738579+0000","last_fullsized":"2023-11-24T06:18:51.738579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892835+0000","last_change":"2023-11-24T06:18:50.035496+0000","last_active":"2023-11-24T06:18:51.892835+0000","last_peered":"2023-11-24T06:18:51.892835+0000","last_clean":"2023-11-24T06:18:51.892835+0000","last_became_active":"2023-11-24T06:18:50.035172+0000","last_became_peered":"2023-11-24T06:18:50.035172+0000","last_unstale":"2023-11-24T06:18:51.892835+0000","last_undegraded":"2023-11-24T06:18:51.892835+0000","last_fullsized":"2023-11-24T06:18:51.892835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:39.141407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738707+0000","last_change":"2023-11-24T06:18:49.012031+0000","last_active":"2023-11-24T06:18:51.738707+0000","last_peered":"2023-11-24T06:18:51.738707+0000","last_clean":"2023-11-24T06:18:51.738707+0000","last_became_active":"2023-11-24T06:18:49.011741+0000","last_became_peered":"2023-11-24T06:18:49.011741+0000","last_unstale":"2023-11-24T06:18:51.738707+0000","last_undegraded":"2023-11-24T06:18:51.738707+0000","last_fullsized":"2023-11-24T06:18:51.738707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.404093+0000","last_change":"2023-11-24T06:18:50.035543+0000","last_active":"2023-11-24T06:18:52.404093+0000","last_peered":"2023-11-24T06:18:52.404093+0000","last_clean":"2023-11-24T06:18:52.404093+0000","last_became_active":"2023-11-24T06:18:50.035269+0000","last_became_peered":"2023-11-24T06:18:50.035269+0000","last_unstale":"2023-11-24T06:18:52.404093+0000","last_undegraded":"2023-11-24T06:18:52.404093+0000","last_fullsized":"2023-11-24T06:18:52.404093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021480+0000","last_change":"2023-11-24T06:18:49.006503+0000","last_active":"2023-11-24T06:18:51.021480+0000","last_peered":"2023-11-24T06:18:51.021480+0000","last_clean":"2023-11-24T06:18:51.021480+0000","last_became_active":"2023-11-24T06:18:49.006176+0000","last_became_peered":"2023-11-24T06:18:49.006176+0000","last_unstale":"2023-11-24T06:18:51.021480+0000","last_undegraded":"2023-11-24T06:18:51.021480+0000","last_fullsized":"2023-11-24T06:18:51.021480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:33.478815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023461+0000","last_change":"2023-11-24T06:18:50.023420+0000","last_active":"2023-11-24T06:18:51.023461+0000","last_peered":"2023-11-24T06:18:51.023461+0000","last_clean":"2023-11-24T06:18:51.023461+0000","last_became_active":"2023-11-24T06:18:50.022722+0000","last_became_peered":"2023-11-24T06:18:50.022722+0000","last_unstale":"2023-11-24T06:18:51.023461+0000","last_undegraded":"2023-11-24T06:18:51.023461+0000","last_fullsized":"2023-11-24T06:18:51.023461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:24.709998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738477+0000","last_change":"2023-11-24T06:18:49.016525+0000","last_active":"2023-11-24T06:18:51.738477+0000","last_peered":"2023-11-24T06:18:51.738477+0000","last_clean":"2023-11-24T06:18:51.738477+0000","last_became_active":"2023-11-24T06:18:49.016259+0000","last_became_peered":"2023-11-24T06:18:49.016259+0000","last_unstale":"2023-11-24T06:18:51.738477+0000","last_undegraded":"2023-11-24T06:18:51.738477+0000","last_fullsized":"2023-11-24T06:18:51.738477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:13.224297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023938+0000","last_change":"2023-11-24T06:18:50.024377+0000","last_active":"2023-11-24T06:18:51.023938+0000","last_peered":"2023-11-24T06:18:51.023938+0000","last_clean":"2023-11-24T06:18:51.023938+0000","last_became_active":"2023-11-24T06:18:50.024123+0000","last_became_peered":"2023-11-24T06:18:50.024123+0000","last_unstale":"2023-11-24T06:18:51.023938+0000","last_undegraded":"2023-11-24T06:18:51.023938+0000","last_fullsized":"2023-11-24T06:18:51.023938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:28:39.352489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.735753+0000","last_change":"2023-11-24T06:18:49.009176+0000","last_active":"2023-11-24T06:18:51.735753+0000","last_peered":"2023-11-24T06:18:51.735753+0000","last_clean":"2023-11-24T06:18:51.735753+0000","last_became_active":"2023-11-24T06:18:49.008875+0000","last_became_peered":"2023-11-24T06:18:49.008875+0000","last_unstale":"2023-11-24T06:18:51.735753+0000","last_undegraded":"2023-11-24T06:18:51.735753+0000","last_fullsized":"2023-11-24T06:18:51.735753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403625+0000","last_change":"2023-11-24T06:18:50.032393+0000","last_active":"2023-11-24T06:18:52.403625+0000","last_peered":"2023-11-24T06:18:52.403625+0000","last_clean":"2023-11-24T06:18:52.403625+0000","last_became_active":"2023-11-24T06:18:50.032145+0000","last_became_peered":"2023-11-24T06:18:50.032145+0000","last_unstale":"2023-11-24T06:18:52.403625+0000","last_undegraded":"2023-11-24T06:18:52.403625+0000","last_fullsized":"2023-11-24T06:18:52.403625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:34.485750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738587+0000","last_change":"2023-11-24T06:18:49.015579+0000","last_active":"2023-11-24T06:18:51.738587+0000","last_peered":"2023-11-24T06:18:51.738587+0000","last_clean":"2023-11-24T06:18:51.738587+0000","last_became_active":"2023-11-24T06:18:49.015329+0000","last_became_peered":"2023-11-24T06:18:49.015329+0000","last_unstale":"2023-11-24T06:18:51.738587+0000","last_undegraded":"2023-11-24T06:18:51.738587+0000","last_fullsized":"2023-11-24T06:18:51.738587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:37.250753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024128+0000","last_change":"2023-11-24T06:18:50.021964+0000","last_active":"2023-11-24T06:18:51.024128+0000","last_peered":"2023-11-24T06:18:51.024128+0000","last_clean":"2023-11-24T06:18:51.024128+0000","last_became_active":"2023-11-24T06:18:50.021678+0000","last_became_peered":"2023-11-24T06:18:50.021678+0000","last_unstale":"2023-11-24T06:18:51.024128+0000","last_undegraded":"2023-11-24T06:18:51.024128+0000","last_fullsized":"2023-11-24T06:18:51.024128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:32:20.935789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738432+0000","last_change":"2023-11-24T06:18:49.007859+0000","last_active":"2023-11-24T06:18:51.738432+0000","last_peered":"2023-11-24T06:18:51.738432+0000","last_clean":"2023-11-24T06:18:51.738432+0000","last_became_active":"2023-11-24T06:18:49.007569+0000","last_became_peered":"2023-11-24T06:18:49.007569+0000","last_unstale":"2023-11-24T06:18:51.738432+0000","last_undegraded":"2023-11-24T06:18:51.738432+0000","last_fullsized":"2023-11-24T06:18:51.738432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.327901+0000","last_change":"2023-11-24T06:18:50.033953+0000","last_active":"2023-11-24T06:18:52.327901+0000","last_peered":"2023-11-24T06:18:52.327901+0000","last_clean":"2023-11-24T06:18:52.327901+0000","last_became_active":"2023-11-24T06:18:50.033664+0000","last_became_peered":"2023-11-24T06:18:50.033664+0000","last_unstale":"2023-11-24T06:18:52.327901+0000","last_undegraded":"2023-11-24T06:18:52.327901+0000","last_fullsized":"2023-11-24T06:18:52.327901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:55.342818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745781+0000","last_change":"2023-11-24T06:18:49.009433+0000","last_active":"2023-11-24T06:18:51.745781+0000","last_peered":"2023-11-24T06:18:51.745781+0000","last_clean":"2023-11-24T06:18:51.745781+0000","last_became_active":"2023-11-24T06:18:49.009133+0000","last_became_peered":"2023-11-24T06:18:49.009133+0000","last_unstale":"2023-11-24T06:18:51.745781+0000","last_undegraded":"2023-11-24T06:18:51.745781+0000","last_fullsized":"2023-11-24T06:18:51.745781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023055+0000","last_change":"2023-11-24T06:18:50.023048+0000","last_active":"2023-11-24T06:18:51.023055+0000","last_peered":"2023-11-24T06:18:51.023055+0000","last_clean":"2023-11-24T06:18:51.023055+0000","last_became_active":"2023-11-24T06:18:50.022736+0000","last_became_peered":"2023-11-24T06:18:50.022736+0000","last_unstale":"2023-11-24T06:18:51.023055+0000","last_undegraded":"2023-11-24T06:18:51.023055+0000","last_fullsized":"2023-11-24T06:18:51.023055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:14.887594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023699+0000","last_change":"2023-11-24T06:18:49.012343+0000","last_active":"2023-11-24T06:18:51.023699+0000","last_peered":"2023-11-24T06:18:51.023699+0000","last_clean":"2023-11-24T06:18:51.023699+0000","last_became_active":"2023-11-24T06:18:49.012078+0000","last_became_peered":"2023-11-24T06:18:49.012078+0000","last_unstale":"2023-11-24T06:18:51.023699+0000","last_undegraded":"2023-11-24T06:18:51.023699+0000","last_fullsized":"2023-11-24T06:18:51.023699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024206+0000","last_change":"2023-11-24T06:18:50.022294+0000","last_active":"2023-11-24T06:18:51.024206+0000","last_peered":"2023-11-24T06:18:51.024206+0000","last_clean":"2023-11-24T06:18:51.024206+0000","last_became_active":"2023-11-24T06:18:50.021806+0000","last_became_peered":"2023-11-24T06:18:50.021806+0000","last_unstale":"2023-11-24T06:18:51.024206+0000","last_undegraded":"2023-11-24T06:18:51.024206+0000","last_fullsized":"2023-11-24T06:18:51.024206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:03.577619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021419+0000","last_change":"2023-11-24T06:18:49.010001+0000","last_active":"2023-11-24T06:18:51.021419+0000","last_peered":"2023-11-24T06:18:51.021419+0000","last_clean":"2023-11-24T06:18:51.021419+0000","last_became_active":"2023-11-24T06:18:49.009729+0000","last_became_peered":"2023-11-24T06:18:49.009729+0000","last_unstale":"2023-11-24T06:18:51.021419+0000","last_undegraded":"2023-11-24T06:18:51.021419+0000","last_fullsized":"2023-11-24T06:18:51.021419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:24.571482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892753+0000","last_change":"2023-11-24T06:18:50.035809+0000","last_active":"2023-11-24T06:18:51.892753+0000","last_peered":"2023-11-24T06:18:51.892753+0000","last_clean":"2023-11-24T06:18:51.892753+0000","last_became_active":"2023-11-24T06:18:50.033211+0000","last_became_peered":"2023-11-24T06:18:50.033211+0000","last_unstale":"2023-11-24T06:18:51.892753+0000","last_undegraded":"2023-11-24T06:18:51.892753+0000","last_fullsized":"2023-11-24T06:18:51.892753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:33:00.547188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738562+0000","last_change":"2023-11-24T06:18:49.004923+0000","last_active":"2023-11-24T06:18:51.738562+0000","last_peered":"2023-11-24T06:18:51.738562+0000","last_clean":"2023-11-24T06:18:51.738562+0000","last_became_active":"2023-11-24T06:18:49.004601+0000","last_became_peered":"2023-11-24T06:18:49.004601+0000","last_unstale":"2023-11-24T06:18:51.738562+0000","last_undegraded":"2023-11-24T06:18:51.738562+0000","last_fullsized":"2023-11-24T06:18:51.738562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:42.514481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023365+0000","last_change":"2023-11-24T06:18:50.026574+0000","last_active":"2023-11-24T06:18:51.023365+0000","last_peered":"2023-11-24T06:18:51.023365+0000","last_clean":"2023-11-24T06:18:51.023365+0000","last_became_active":"2023-11-24T06:18:50.026028+0000","last_became_peered":"2023-11-24T06:18:50.026028+0000","last_unstale":"2023-11-24T06:18:51.023365+0000","last_undegraded":"2023-11-24T06:18:51.023365+0000","last_fullsized":"2023-11-24T06:18:51.023365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021109+0000","last_change":"2023-11-24T06:18:50.027544+0000","last_active":"2023-11-24T06:18:51.021109+0000","last_peered":"2023-11-24T06:18:51.021109+0000","last_clean":"2023-11-24T06:18:51.021109+0000","last_became_active":"2023-11-24T06:18:50.027272+0000","last_became_peered":"2023-11-24T06:18:50.027272+0000","last_unstale":"2023-11-24T06:18:51.021109+0000","last_undegraded":"2023-11-24T06:18:51.021109+0000","last_fullsized":"2023-11-24T06:18:51.021109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745828+0000","last_change":"2023-11-24T06:18:49.015756+0000","last_active":"2023-11-24T06:18:51.745828+0000","last_peered":"2023-11-24T06:18:51.745828+0000","last_clean":"2023-11-24T06:18:51.745828+0000","last_became_active":"2023-11-24T06:18:49.015446+0000","last_became_peered":"2023-11-24T06:18:49.015446+0000","last_unstale":"2023-11-24T06:18:51.745828+0000","last_undegraded":"2023-11-24T06:18:51.745828+0000","last_fullsized":"2023-11-24T06:18:51.745828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:24.571482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403228+0000","last_change":"2023-11-24T06:18:50.040018+0000","last_active":"2023-11-24T06:18:52.403228+0000","last_peered":"2023-11-24T06:18:52.403228+0000","last_clean":"2023-11-24T06:18:52.403228+0000","last_became_active":"2023-11-24T06:18:50.039786+0000","last_became_peered":"2023-11-24T06:18:50.039786+0000","last_unstale":"2023-11-24T06:18:52.403228+0000","last_undegraded":"2023-11-24T06:18:52.403228+0000","last_fullsized":"2023-11-24T06:18:52.403228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738619+0000","last_change":"2023-11-24T06:18:49.009457+0000","last_active":"2023-11-24T06:18:51.738619+0000","last_peered":"2023-11-24T06:18:51.738619+0000","last_clean":"2023-11-24T06:18:51.738619+0000","last_became_active":"2023-11-24T06:18:49.008932+0000","last_became_peered":"2023-11-24T06:18:49.008932+0000","last_unstale":"2023-11-24T06:18:51.738619+0000","last_undegraded":"2023-11-24T06:18:51.738619+0000","last_fullsized":"2023-11-24T06:18:51.738619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.009997+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:18:50.009997+0000","last_peered":"2023-11-24T06:18:50.009997+0000","last_clean":"2023-11-24T06:18:50.009997+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:18:50.009997+0000","last_undegraded":"2023-11-24T06:18:50.009997+0000","last_fullsized":"2023-11-24T06:18:50.009997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403625+0000","last_change":"2023-11-24T06:18:50.027632+0000","last_active":"2023-11-24T06:18:52.403625+0000","last_peered":"2023-11-24T06:18:52.403625+0000","last_clean":"2023-11-24T06:18:52.403625+0000","last_became_active":"2023-11-24T06:18:50.027344+0000","last_became_peered":"2023-11-24T06:18:50.027344+0000","last_unstale":"2023-11-24T06:18:52.403625+0000","last_undegraded":"2023-11-24T06:18:52.403625+0000","last_fullsized":"2023-11-24T06:18:52.403625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:56:59.481984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022810+0000","last_change":"2023-11-24T06:18:49.006996+0000","last_active":"2023-11-24T06:18:51.022810+0000","last_peered":"2023-11-24T06:18:51.022810+0000","last_clean":"2023-11-24T06:18:51.022810+0000","last_became_active":"2023-11-24T06:18:49.006652+0000","last_became_peered":"2023-11-24T06:18:49.006652+0000","last_unstale":"2023-11-24T06:18:51.022810+0000","last_undegraded":"2023-11-24T06:18:51.022810+0000","last_fullsized":"2023-11-24T06:18:51.022810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:01.167887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024055+0000","last_change":"2023-11-24T06:18:49.012066+0000","last_active":"2023-11-24T06:18:51.024055+0000","last_peered":"2023-11-24T06:18:51.024055+0000","last_clean":"2023-11-24T06:18:51.024055+0000","last_became_active":"2023-11-24T06:18:49.011795+0000","last_became_peered":"2023-11-24T06:18:49.011795+0000","last_unstale":"2023-11-24T06:18:51.024055+0000","last_undegraded":"2023-11-24T06:18:51.024055+0000","last_fullsized":"2023-11-24T06:18:51.024055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:22.850843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021314+0000","last_change":"2023-11-24T06:18:50.021739+0000","last_active":"2023-11-24T06:18:51.021314+0000","last_peered":"2023-11-24T06:18:51.021314+0000","last_clean":"2023-11-24T06:18:51.021314+0000","last_became_active":"2023-11-24T06:18:50.021452+0000","last_became_peered":"2023-11-24T06:18:50.021452+0000","last_unstale":"2023-11-24T06:18:51.021314+0000","last_undegraded":"2023-11-24T06:18:51.021314+0000","last_fullsized":"2023-11-24T06:18:51.021314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745909+0000","last_change":"2023-11-24T06:18:49.012066+0000","last_active":"2023-11-24T06:18:51.745909+0000","last_peered":"2023-11-24T06:18:51.745909+0000","last_clean":"2023-11-24T06:18:51.745909+0000","last_became_active":"2023-11-24T06:18:49.011748+0000","last_became_peered":"2023-11-24T06:18:49.011748+0000","last_unstale":"2023-11-24T06:18:51.745909+0000","last_undegraded":"2023-11-24T06:18:51.745909+0000","last_fullsized":"2023-11-24T06:18:51.745909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.404100+0000","last_change":"2023-11-24T06:18:50.033206+0000","last_active":"2023-11-24T06:18:52.404100+0000","last_peered":"2023-11-24T06:18:52.404100+0000","last_clean":"2023-11-24T06:18:52.404100+0000","last_became_active":"2023-11-24T06:18:50.032878+0000","last_became_peered":"2023-11-24T06:18:50.032878+0000","last_unstale":"2023-11-24T06:18:52.404100+0000","last_undegraded":"2023-11-24T06:18:52.404100+0000","last_fullsized":"2023-11-24T06:18:52.404100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:28:39.352489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738568+0000","last_change":"2023-11-24T06:18:49.011356+0000","last_active":"2023-11-24T06:18:51.738568+0000","last_peered":"2023-11-24T06:18:51.738568+0000","last_clean":"2023-11-24T06:18:51.738568+0000","last_became_active":"2023-11-24T06:18:49.010903+0000","last_became_peered":"2023-11-24T06:18:49.010903+0000","last_unstale":"2023-11-24T06:18:51.738568+0000","last_undegraded":"2023-11-24T06:18:51.738568+0000","last_fullsized":"2023-11-24T06:18:51.738568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403395+0000","last_change":"2023-11-24T06:18:50.024110+0000","last_active":"2023-11-24T06:18:52.403395+0000","last_peered":"2023-11-24T06:18:52.403395+0000","last_clean":"2023-11-24T06:18:52.403395+0000","last_became_active":"2023-11-24T06:18:50.023801+0000","last_became_peered":"2023-11-24T06:18:50.023801+0000","last_unstale":"2023-11-24T06:18:52.403395+0000","last_undegraded":"2023-11-24T06:18:52.403395+0000","last_fullsized":"2023-11-24T06:18:52.403395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:54.282518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022707+0000","last_change":"2023-11-24T06:18:49.010428+0000","last_active":"2023-11-24T06:18:51.022707+0000","last_peered":"2023-11-24T06:18:51.022707+0000","last_clean":"2023-11-24T06:18:51.022707+0000","last_became_active":"2023-11-24T06:18:49.010124+0000","last_became_peered":"2023-11-24T06:18:49.010124+0000","last_unstale":"2023-11-24T06:18:51.022707+0000","last_undegraded":"2023-11-24T06:18:51.022707+0000","last_fullsized":"2023-11-24T06:18:51.022707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023378+0000","last_change":"2023-11-24T06:18:50.022092+0000","last_active":"2023-11-24T06:18:51.023378+0000","last_peered":"2023-11-24T06:18:51.023378+0000","last_clean":"2023-11-24T06:18:51.023378+0000","last_became_active":"2023-11-24T06:18:50.021808+0000","last_became_peered":"2023-11-24T06:18:50.021808+0000","last_unstale":"2023-11-24T06:18:51.023378+0000","last_undegraded":"2023-11-24T06:18:51.023378+0000","last_fullsized":"2023-11-24T06:18:51.023378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:21.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021058+0000","last_change":"2023-11-24T06:18:49.007474+0000","last_active":"2023-11-24T06:18:51.021058+0000","last_peered":"2023-11-24T06:18:51.021058+0000","last_clean":"2023-11-24T06:18:51.021058+0000","last_became_active":"2023-11-24T06:18:49.007228+0000","last_became_peered":"2023-11-24T06:18:49.007228+0000","last_unstale":"2023-11-24T06:18:51.021058+0000","last_undegraded":"2023-11-24T06:18:51.021058+0000","last_fullsized":"2023-11-24T06:18:51.021058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403034+0000","last_change":"2023-11-24T06:18:50.025141+0000","last_active":"2023-11-24T06:18:52.403034+0000","last_peered":"2023-11-24T06:18:52.403034+0000","last_clean":"2023-11-24T06:18:52.403034+0000","last_became_active":"2023-11-24T06:18:50.024859+0000","last_became_peered":"2023-11-24T06:18:50.024859+0000","last_unstale":"2023-11-24T06:18:52.403034+0000","last_undegraded":"2023-11-24T06:18:52.403034+0000","last_fullsized":"2023-11-24T06:18:52.403034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021233+0000","last_change":"2023-11-24T06:18:49.015615+0000","last_active":"2023-11-24T06:18:51.021233+0000","last_peered":"2023-11-24T06:18:51.021233+0000","last_clean":"2023-11-24T06:18:51.021233+0000","last_became_active":"2023-11-24T06:18:49.015320+0000","last_became_peered":"2023-11-24T06:18:49.015320+0000","last_unstale":"2023-11-24T06:18:51.021233+0000","last_undegraded":"2023-11-24T06:18:51.021233+0000","last_fullsized":"2023-11-24T06:18:51.021233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403348+0000","last_change":"2023-11-24T06:18:50.025429+0000","last_active":"2023-11-24T06:18:52.403348+0000","last_peered":"2023-11-24T06:18:52.403348+0000","last_clean":"2023-11-24T06:18:52.403348+0000","last_became_active":"2023-11-24T06:18:50.024972+0000","last_became_peered":"2023-11-24T06:18:50.024972+0000","last_unstale":"2023-11-24T06:18:52.403348+0000","last_undegraded":"2023-11-24T06:18:52.403348+0000","last_fullsized":"2023-11-24T06:18:52.403348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:48.527966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024261+0000","last_change":"2023-11-24T06:18:49.007100+0000","last_active":"2023-11-24T06:18:51.024261+0000","last_peered":"2023-11-24T06:18:51.024261+0000","last_clean":"2023-11-24T06:18:51.024261+0000","last_became_active":"2023-11-24T06:18:49.006577+0000","last_became_peered":"2023-11-24T06:18:49.006577+0000","last_unstale":"2023-11-24T06:18:51.024261+0000","last_undegraded":"2023-11-24T06:18:51.024261+0000","last_fullsized":"2023-11-24T06:18:51.024261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328023+0000","last_change":"2023-11-24T06:18:50.032437+0000","last_active":"2023-11-24T06:18:52.328023+0000","last_peered":"2023-11-24T06:18:52.328023+0000","last_clean":"2023-11-24T06:18:52.328023+0000","last_became_active":"2023-11-24T06:18:50.032082+0000","last_became_peered":"2023-11-24T06:18:50.032082+0000","last_unstale":"2023-11-24T06:18:52.328023+0000","last_undegraded":"2023-11-24T06:18:52.328023+0000","last_fullsized":"2023-11-24T06:18:52.328023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745848+0000","last_change":"2023-11-24T06:18:49.011290+0000","last_active":"2023-11-24T06:18:51.745848+0000","last_peered":"2023-11-24T06:18:51.745848+0000","last_clean":"2023-11-24T06:18:51.745848+0000","last_became_active":"2023-11-24T06:18:49.010889+0000","last_became_peered":"2023-11-24T06:18:49.010889+0000","last_unstale":"2023-11-24T06:18:51.745848+0000","last_undegraded":"2023-11-24T06:18:51.745848+0000","last_fullsized":"2023-11-24T06:18:51.745848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024098+0000","last_change":"2023-11-24T06:18:50.036167+0000","last_active":"2023-11-24T06:18:51.024098+0000","last_peered":"2023-11-24T06:18:51.024098+0000","last_clean":"2023-11-24T06:18:51.024098+0000","last_became_active":"2023-11-24T06:18:50.035904+0000","last_became_peered":"2023-11-24T06:18:50.035904+0000","last_unstale":"2023-11-24T06:18:51.024098+0000","last_undegraded":"2023-11-24T06:18:51.024098+0000","last_fullsized":"2023-11-24T06:18:51.024098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:33:00.547188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021448+0000","last_change":"2023-11-24T06:18:49.006925+0000","last_active":"2023-11-24T06:18:51.021448+0000","last_peered":"2023-11-24T06:18:51.021448+0000","last_clean":"2023-11-24T06:18:51.021448+0000","last_became_active":"2023-11-24T06:18:49.006482+0000","last_became_peered":"2023-11-24T06:18:49.006482+0000","last_unstale":"2023-11-24T06:18:51.021448+0000","last_undegraded":"2023-11-24T06:18:51.021448+0000","last_fullsized":"2023-11-24T06:18:51.021448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:40.335012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892543+0000","last_change":"2023-11-24T06:18:50.039417+0000","last_active":"2023-11-24T06:18:51.892543+0000","last_peered":"2023-11-24T06:18:51.892543+0000","last_clean":"2023-11-24T06:18:51.892543+0000","last_became_active":"2023-11-24T06:18:50.031461+0000","last_became_peered":"2023-11-24T06:18:50.031461+0000","last_unstale":"2023-11-24T06:18:51.892543+0000","last_undegraded":"2023-11-24T06:18:51.892543+0000","last_fullsized":"2023-11-24T06:18:51.892543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:28:39.352489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745887+0000","last_change":"2023-11-24T06:18:49.015786+0000","last_active":"2023-11-24T06:18:51.745887+0000","last_peered":"2023-11-24T06:18:51.745887+0000","last_clean":"2023-11-24T06:18:51.745887+0000","last_became_active":"2023-11-24T06:18:49.015506+0000","last_became_peered":"2023-11-24T06:18:49.015506+0000","last_unstale":"2023-11-24T06:18:51.745887+0000","last_undegraded":"2023-11-24T06:18:51.745887+0000","last_fullsized":"2023-11-24T06:18:51.745887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:01.167887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024358+0000","last_change":"2023-11-24T06:18:50.023402+0000","last_active":"2023-11-24T06:18:51.024358+0000","last_peered":"2023-11-24T06:18:51.024358+0000","last_clean":"2023-11-24T06:18:51.024358+0000","last_became_active":"2023-11-24T06:18:50.023117+0000","last_became_peered":"2023-11-24T06:18:50.023117+0000","last_unstale":"2023-11-24T06:18:51.024358+0000","last_undegraded":"2023-11-24T06:18:51.024358+0000","last_fullsized":"2023-11-24T06:18:51.024358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:54.022789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.327886+0000","last_change":"2023-11-24T06:18:50.032212+0000","last_active":"2023-11-24T06:18:52.327886+0000","last_peered":"2023-11-24T06:18:52.327886+0000","last_clean":"2023-11-24T06:18:52.327886+0000","last_became_active":"2023-11-24T06:18:50.031933+0000","last_became_peered":"2023-11-24T06:18:50.031933+0000","last_unstale":"2023-11-24T06:18:52.327886+0000","last_undegraded":"2023-11-24T06:18:52.327886+0000","last_fullsized":"2023-11-24T06:18:52.327886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:56:59.481984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403264+0000","last_change":"2023-11-24T06:18:50.037725+0000","last_active":"2023-11-24T06:18:52.403264+0000","last_peered":"2023-11-24T06:18:52.403264+0000","last_clean":"2023-11-24T06:18:52.403264+0000","last_became_active":"2023-11-24T06:18:50.037471+0000","last_became_peered":"2023-11-24T06:18:50.037471+0000","last_unstale":"2023-11-24T06:18:52.403264+0000","last_undegraded":"2023-11-24T06:18:52.403264+0000","last_fullsized":"2023-11-24T06:18:52.403264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:24.709998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892512+0000","last_change":"2023-11-24T06:18:50.038973+0000","last_active":"2023-11-24T06:18:51.892512+0000","last_peered":"2023-11-24T06:18:51.892512+0000","last_clean":"2023-11-24T06:18:51.892512+0000","last_became_active":"2023-11-24T06:18:50.038345+0000","last_became_peered":"2023-11-24T06:18:50.038345+0000","last_unstale":"2023-11-24T06:18:51.892512+0000","last_undegraded":"2023-11-24T06:18:51.892512+0000","last_fullsized":"2023-11-24T06:18:51.892512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:02:29.652618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403787+0000","last_change":"2023-11-24T06:18:50.024376+0000","last_active":"2023-11-24T06:18:52.403787+0000","last_peered":"2023-11-24T06:18:52.403787+0000","last_clean":"2023-11-24T06:18:52.403787+0000","last_became_active":"2023-11-24T06:18:50.024118+0000","last_became_peered":"2023-11-24T06:18:50.024118+0000","last_unstale":"2023-11-24T06:18:52.403787+0000","last_undegraded":"2023-11-24T06:18:52.403787+0000","last_fullsized":"2023-11-24T06:18:52.403787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403334+0000","last_change":"2023-11-24T06:18:50.028334+0000","last_active":"2023-11-24T06:18:52.403334+0000","last_peered":"2023-11-24T06:18:52.403334+0000","last_clean":"2023-11-24T06:18:52.403334+0000","last_became_active":"2023-11-24T06:18:50.028091+0000","last_became_peered":"2023-11-24T06:18:50.028091+0000","last_unstale":"2023-11-24T06:18:52.403334+0000","last_undegraded":"2023-11-24T06:18:52.403334+0000","last_fullsized":"2023-11-24T06:18:52.403334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:59.327185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021380+0000","last_change":"2023-11-24T06:18:50.025563+0000","last_active":"2023-11-24T06:18:51.021380+0000","last_peered":"2023-11-24T06:18:51.021380+0000","last_clean":"2023-11-24T06:18:51.021380+0000","last_became_active":"2023-11-24T06:18:50.025276+0000","last_became_peered":"2023-11-24T06:18:50.025276+0000","last_unstale":"2023-11-24T06:18:51.021380+0000","last_undegraded":"2023-11-24T06:18:51.021380+0000","last_fullsized":"2023-11-24T06:18:51.021380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328320+0000","last_change":"2023-11-24T06:18:50.032672+0000","last_active":"2023-11-24T06:18:52.328320+0000","last_peered":"2023-11-24T06:18:52.328320+0000","last_clean":"2023-11-24T06:18:52.328320+0000","last_became_active":"2023-11-24T06:18:50.032400+0000","last_became_peered":"2023-11-24T06:18:50.032400+0000","last_unstale":"2023-11-24T06:18:52.328320+0000","last_undegraded":"2023-11-24T06:18:52.328320+0000","last_fullsized":"2023-11-24T06:18:52.328320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.404010+0000","last_change":"2023-11-24T06:18:50.027189+0000","last_active":"2023-11-24T06:18:52.404010+0000","last_peered":"2023-11-24T06:18:52.404010+0000","last_clean":"2023-11-24T06:18:52.404010+0000","last_became_active":"2023-11-24T06:18:50.026893+0000","last_became_peered":"2023-11-24T06:18:50.026893+0000","last_unstale":"2023-11-24T06:18:52.404010+0000","last_undegraded":"2023-11-24T06:18:52.404010+0000","last_fullsized":"2023-11-24T06:18:52.404010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:02:29.652618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892394+0000","last_change":"2023-11-24T06:18:50.031274+0000","last_active":"2023-11-24T06:18:51.892394+0000","last_peered":"2023-11-24T06:18:51.892394+0000","last_clean":"2023-11-24T06:18:51.892394+0000","last_became_active":"2023-11-24T06:18:50.030997+0000","last_became_peered":"2023-11-24T06:18:50.030997+0000","last_unstale":"2023-11-24T06:18:51.892394+0000","last_undegraded":"2023-11-24T06:18:51.892394+0000","last_fullsized":"2023-11-24T06:18:51.892394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:59.327185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403094+0000","last_change":"2023-11-24T06:18:50.027675+0000","last_active":"2023-11-24T06:18:52.403094+0000","last_peered":"2023-11-24T06:18:52.403094+0000","last_clean":"2023-11-24T06:18:52.403094+0000","last_became_active":"2023-11-24T06:18:50.027402+0000","last_became_peered":"2023-11-24T06:18:50.027402+0000","last_unstale":"2023-11-24T06:18:52.403094+0000","last_undegraded":"2023-11-24T06:18:52.403094+0000","last_fullsized":"2023-11-24T06:18:52.403094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892381+0000","last_change":"2023-11-24T06:18:50.039174+0000","last_active":"2023-11-24T06:18:51.892381+0000","last_peered":"2023-11-24T06:18:51.892381+0000","last_clean":"2023-11-24T06:18:51.892381+0000","last_became_active":"2023-11-24T06:18:50.038671+0000","last_became_peered":"2023-11-24T06:18:50.038671+0000","last_unstale":"2023-11-24T06:18:51.892381+0000","last_undegraded":"2023-11-24T06:18:51.892381+0000","last_fullsized":"2023-11-24T06:18:51.892381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:48.527966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403711+0000","last_change":"2023-11-24T06:18:50.032515+0000","last_active":"2023-11-24T06:18:52.403711+0000","last_peered":"2023-11-24T06:18:52.403711+0000","last_clean":"2023-11-24T06:18:52.403711+0000","last_became_active":"2023-11-24T06:18:50.032238+0000","last_became_peered":"2023-11-24T06:18:50.032238+0000","last_unstale":"2023-11-24T06:18:52.403711+0000","last_undegraded":"2023-11-24T06:18:52.403711+0000","last_fullsized":"2023-11-24T06:18:52.403711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:55.342818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328180+0000","last_change":"2023-11-24T06:18:50.035460+0000","last_active":"2023-11-24T06:18:52.328180+0000","last_peered":"2023-11-24T06:18:52.328180+0000","last_clean":"2023-11-24T06:18:52.328180+0000","last_became_active":"2023-11-24T06:18:50.031929+0000","last_became_peered":"2023-11-24T06:18:50.031929+0000","last_unstale":"2023-11-24T06:18:52.328180+0000","last_undegraded":"2023-11-24T06:18:52.328180+0000","last_fullsized":"2023-11-24T06:18:52.328180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.423}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.554}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0489999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0920000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.198}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:18:58.748 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:18:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.902+0000 7fa547fef700 1 Processor -- start 2023-11-24T06:18:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.902+0000 7fa547fef700 1 -- start start 2023-11-24T06:18:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa547fef700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015ddd0 0x7fa5400b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa547fef700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa54015e1b0 0x7fa5400cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa547fef700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5400d1b30 0x7fa5400d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa547fef700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400d2500 con 0x7fa5400d1b30 2023-11-24T06:18:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa547fef700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400b1d40 con 0x7fa54015e1b0 2023-11-24T06:18:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa547fef700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400b28c0 con 0x7fa54015ddd0 2023-11-24T06:18:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa54658c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5400d1b30 0x7fa5400d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa54658c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5400d1b30 0x7fa5400d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35480/0 (socket says 172.21.15.89:35480) 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.903+0000 7fa54658c700 1 -- 172.21.15.89:0/3436444663 learned_addr learned my addr 172.21.15.89:0/3436444663 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa54558a700 1 --2- 172.21.15.89:0/3436444663 >> v2:172.21.15.123:3300/0 conn(0x7fa54015e1b0 0x7fa5400cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa545d8b700 1 --2- 172.21.15.89:0/3436444663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015ddd0 0x7fa5400b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa54658c700 1 -- 172.21.15.89:0/3436444663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015ddd0 msgr2=0x7fa5400b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa54658c700 1 --2- 172.21.15.89:0/3436444663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015ddd0 0x7fa5400b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa54658c700 1 -- 172.21.15.89:0/3436444663 >> v2:172.21.15.123:3300/0 conn(0x7fa54015e1b0 msgr2=0x7fa5400cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa54658c700 1 --2- 172.21.15.89:0/3436444663 >> v2:172.21.15.123:3300/0 conn(0x7fa54015e1b0 0x7fa5400cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa54658c700 1 -- 172.21.15.89:0/3436444663 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400b1f90 con 0x7fa5400d1b30 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa54558a700 1 --2- 172.21.15.89:0/3436444663 >> v2:172.21.15.123:3300/0 conn(0x7fa54015e1b0 0x7fa5400cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa54658c700 1 --2- 172.21.15.89:0/3436444663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5400d1b30 0x7fa5400d1fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa5400cf7a0 tx=0x7fa53800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dcccb9ae9659936 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:58.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa544d89700 1 -- 172.21.15.89:0/3436444663 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53800e070 con 0x7fa5400d1b30 2023-11-24T06:18:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa544d89700 1 -- 172.21.15.89:0/3436444663 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53800b9c0 con 0x7fa5400d1b30 2023-11-24T06:18:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa544d89700 1 -- 172.21.15.89:0/3436444663 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53800bd70 con 0x7fa5400d1b30 2023-11-24T06:18:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa547fef700 1 -- 172.21.15.89:0/3436444663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5400d1b30 msgr2=0x7fa5400d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.904+0000 7fa547fef700 1 --2- 172.21.15.89:0/3436444663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5400d1b30 0x7fa5400d1fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa5400cf7a0 tx=0x7fa53800b330 comp rx=0 tx=0).stop 2023-11-24T06:18:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.905+0000 7fa547fef700 1 -- 172.21.15.89:0/3436444663 shutdown_connections 2023-11-24T06:18:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.905+0000 7fa547fef700 1 --2- 172.21.15.89:0/3436444663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5400d1b30 0x7fa5400d1fc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.905+0000 7fa547fef700 1 --2- 172.21.15.89:0/3436444663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015ddd0 0x7fa5400b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.905+0000 7fa547fef700 1 --2- 172.21.15.89:0/3436444663 >> v2:172.21.15.123:3300/0 conn(0x7fa54015e1b0 0x7fa5400cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.905+0000 7fa547fef700 1 -- 172.21.15.89:0/3436444663 >> 172.21.15.89:0/3436444663 conn(0x7fa5400731a0 msgr2=0x7fa54009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.905+0000 7fa547fef700 1 -- 172.21.15.89:0/3436444663 shutdown_connections 2023-11-24T06:18:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.905+0000 7fa547fef700 1 -- 172.21.15.89:0/3436444663 wait complete. 2023-11-24T06:18:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.906+0000 7fa547fef700 1 Processor -- start 2023-11-24T06:18:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.906+0000 7fa547fef700 1 -- start start 2023-11-24T06:18:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa547fef700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5400d1b30 0x7fa54009c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa547fef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa54015ddd0 0x7fa5400b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa547fef700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015e1b0 0x7fa5400afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa547fef700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54014af80 con 0x7fa54015ddd0 2023-11-24T06:18:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa547fef700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400b3260 con 0x7fa5400d1b30 2023-11-24T06:18:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa547fef700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400b3540 con 0x7fa54015e1b0 2023-11-24T06:18:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa54558a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa54015ddd0 0x7fa5400b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa54558a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa54015ddd0 0x7fa5400b7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35484/0 (socket says 172.21.15.89:35484) 2023-11-24T06:18:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa54558a700 1 -- 172.21.15.89:0/3654656880 learned_addr learned my addr 172.21.15.89:0/3654656880 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa54658c700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015e1b0 0x7fa5400afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa545d8b700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5400d1b30 0x7fa54009c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa54558a700 1 -- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015e1b0 msgr2=0x7fa5400afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa54558a700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015e1b0 0x7fa5400afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa54558a700 1 -- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5400d1b30 msgr2=0x7fa54009c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.907+0000 7fa54558a700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5400d1b30 0x7fa54009c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa54558a700 1 -- 172.21.15.89:0/3654656880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538009b30 con 0x7fa54015ddd0 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa54658c700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015e1b0 0x7fa5400afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa545d8b700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5400d1b30 0x7fa54009c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa54558a700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa54015ddd0 0x7fa5400b7210 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00e810 tx=0x7fa53c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:58.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa5367fc700 1 -- 172.21.15.89:0/3654656880 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c00c020 con 0x7fa54015ddd0 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa5367fc700 1 -- 172.21.15.89:0/3654656880 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c004250 con 0x7fa54015ddd0 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5400b2ed0 con 0x7fa54015ddd0 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa5367fc700 1 -- 172.21.15.89:0/3654656880 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c0036a0 con 0x7fa54015ddd0 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.908+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54015a6f0 con 0x7fa54015ddd0 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.909+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa540067f60 con 0x7fa54015ddd0 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.909+0000 7fa5367fc700 1 -- 172.21.15.89:0/3654656880 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa53c003800 con 0x7fa54015ddd0 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.910+0000 7fa5367fc700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa52406aed0 0x7fa52406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.910+0000 7fa545d8b700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa52406aed0 0x7fa52406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:58.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.910+0000 7fa5367fc700 1 -- 172.21.15.89:0/3654656880 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa53c089030 con 0x7fa54015ddd0 2023-11-24T06:18:58.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.910+0000 7fa545d8b700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa52406aed0 0x7fa52406d390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa5300099e0 tx=0x7fa530008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:58.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:58.914+0000 7fa5367fc700 1 -- 172.21.15.89:0/3654656880 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa53c054170 con 0x7fa54015ddd0 2023-11-24T06:18:59.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.100+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5400b2d50 con 0x7fa52406aed0 2023-11-24T06:18:59.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.107+0000 7fa5367fc700 1 -- 172.21.15.89:0/3654656880 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264578 (secure 0 0 0) 0x7fa530009e50 con 0x7fa52406aed0 2023-11-24T06:18:59.114 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:59.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa52406aed0 msgr2=0x7fa52406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa52406aed0 0x7fa52406d390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa5300099e0 tx=0x7fa530008040 comp rx=0 tx=0).stop 2023-11-24T06:18:59.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa54015ddd0 msgr2=0x7fa5400b7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa54015ddd0 0x7fa5400b7210 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00e810 tx=0x7fa53c00ebd0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 shutdown_connections 2023-11-24T06:18:59.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa54015e1b0 0x7fa5400afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa52406aed0 0x7fa52406d390 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5400d1b30 0x7fa54009c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 --2- 172.21.15.89:0/3654656880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa54015ddd0 0x7fa5400b7210 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 >> 172.21.15.89:0/3654656880 conn(0x7fa5400731a0 msgr2=0x7fa54015e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:59.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 shutdown_connections 2023-11-24T06:18:59.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.111+0000 7fa547fef700 1 -- 172.21.15.89:0/3654656880 wait complete. 2023-11-24T06:18:59.119 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:18:59.125 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-24T06:18:57.721114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219104,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755616,"statfs":{"total":773094113280,"available":772869750784,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003267"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024214+0000","last_change":"2023-11-24T06:18:50.033723+0000","last_active":"2023-11-24T06:18:51.024214+0000","last_peered":"2023-11-24T06:18:51.024214+0000","last_clean":"2023-11-24T06:18:51.024214+0000","last_became_active":"2023-11-24T06:18:50.033433+0000","last_became_peered":"2023-11-24T06:18:50.033433+0000","last_unstale":"2023-11-24T06:18:51.024214+0000","last_undegraded":"2023-11-24T06:18:51.024214+0000","last_fullsized":"2023-11-24T06:18:51.024214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:37:49.822079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021557+0000","last_change":"2023-11-24T06:18:50.024643+0000","last_active":"2023-11-24T06:18:51.021557+0000","last_peered":"2023-11-24T06:18:51.021557+0000","last_clean":"2023-11-24T06:18:51.021557+0000","last_became_active":"2023-11-24T06:18:50.024352+0000","last_became_peered":"2023-11-24T06:18:50.024352+0000","last_unstale":"2023-11-24T06:18:51.021557+0000","last_undegraded":"2023-11-24T06:18:51.021557+0000","last_fullsized":"2023-11-24T06:18:51.021557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:59.327185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403187+0000","last_change":"2023-11-24T06:18:50.025637+0000","last_active":"2023-11-24T06:18:52.403187+0000","last_peered":"2023-11-24T06:18:52.403187+0000","last_clean":"2023-11-24T06:18:52.403187+0000","last_became_active":"2023-11-24T06:18:50.025328+0000","last_became_peered":"2023-11-24T06:18:50.025328+0000","last_unstale":"2023-11-24T06:18:52.403187+0000","last_undegraded":"2023-11-24T06:18:52.403187+0000","last_fullsized":"2023-11-24T06:18:52.403187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:41.341947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892759+0000","last_change":"2023-11-24T06:18:50.030667+0000","last_active":"2023-11-24T06:18:51.892759+0000","last_peered":"2023-11-24T06:18:51.892759+0000","last_clean":"2023-11-24T06:18:51.892759+0000","last_became_active":"2023-11-24T06:18:50.030133+0000","last_became_peered":"2023-11-24T06:18:50.030133+0000","last_unstale":"2023-11-24T06:18:51.892759+0000","last_undegraded":"2023-11-24T06:18:51.892759+0000","last_fullsized":"2023-11-24T06:18:51.892759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:21.299524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024116+0000","last_change":"2023-11-24T06:18:50.025413+0000","last_active":"2023-11-24T06:18:51.024116+0000","last_peered":"2023-11-24T06:18:51.024116+0000","last_clean":"2023-11-24T06:18:51.024116+0000","last_became_active":"2023-11-24T06:18:50.025118+0000","last_became_peered":"2023-11-24T06:18:50.025118+0000","last_unstale":"2023-11-24T06:18:51.024116+0000","last_undegraded":"2023-11-24T06:18:51.024116+0000","last_fullsized":"2023-11-24T06:18:51.024116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:21.299524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892331+0000","last_change":"2023-11-24T06:18:50.039892+0000","last_active":"2023-11-24T06:18:51.892331+0000","last_peered":"2023-11-24T06:18:51.892331+0000","last_clean":"2023-11-24T06:18:51.892331+0000","last_became_active":"2023-11-24T06:18:50.039525+0000","last_became_peered":"2023-11-24T06:18:50.039525+0000","last_unstale":"2023-11-24T06:18:51.892331+0000","last_undegraded":"2023-11-24T06:18:51.892331+0000","last_fullsized":"2023-11-24T06:18:51.892331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022843+0000","last_change":"2023-11-24T06:18:50.024213+0000","last_active":"2023-11-24T06:18:51.022843+0000","last_peered":"2023-11-24T06:18:51.022843+0000","last_clean":"2023-11-24T06:18:51.022843+0000","last_became_active":"2023-11-24T06:18:50.023914+0000","last_became_peered":"2023-11-24T06:18:50.023914+0000","last_unstale":"2023-11-24T06:18:51.022843+0000","last_undegraded":"2023-11-24T06:18:51.022843+0000","last_fullsized":"2023-11-24T06:18:51.022843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:48.527966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.402897+0000","last_change":"2023-11-24T06:18:50.037907+0000","last_active":"2023-11-24T06:18:52.402897+0000","last_peered":"2023-11-24T06:18:52.402897+0000","last_clean":"2023-11-24T06:18:52.402897+0000","last_became_active":"2023-11-24T06:18:50.037606+0000","last_became_peered":"2023-11-24T06:18:50.037606+0000","last_unstale":"2023-11-24T06:18:52.402897+0000","last_undegraded":"2023-11-24T06:18:52.402897+0000","last_fullsized":"2023-11-24T06:18:52.402897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:55.342818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024189+0000","last_change":"2023-11-24T06:18:50.035382+0000","last_active":"2023-11-24T06:18:51.024189+0000","last_peered":"2023-11-24T06:18:51.024189+0000","last_clean":"2023-11-24T06:18:51.024189+0000","last_became_active":"2023-11-24T06:18:50.035148+0000","last_became_peered":"2023-11-24T06:18:50.035148+0000","last_unstale":"2023-11-24T06:18:51.024189+0000","last_undegraded":"2023-11-24T06:18:51.024189+0000","last_fullsized":"2023-11-24T06:18:51.024189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:39.141407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328435+0000","last_change":"2023-11-24T06:18:50.037425+0000","last_active":"2023-11-24T06:18:52.328435+0000","last_peered":"2023-11-24T06:18:52.328435+0000","last_clean":"2023-11-24T06:18:52.328435+0000","last_became_active":"2023-11-24T06:18:50.037111+0000","last_became_peered":"2023-11-24T06:18:50.037111+0000","last_unstale":"2023-11-24T06:18:52.328435+0000","last_undegraded":"2023-11-24T06:18:52.328435+0000","last_fullsized":"2023-11-24T06:18:52.328435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328117+0000","last_change":"2023-11-24T06:18:50.036955+0000","last_active":"2023-11-24T06:18:52.328117+0000","last_peered":"2023-11-24T06:18:52.328117+0000","last_clean":"2023-11-24T06:18:52.328117+0000","last_became_active":"2023-11-24T06:18:50.036643+0000","last_became_peered":"2023-11-24T06:18:50.036643+0000","last_unstale":"2023-11-24T06:18:52.328117+0000","last_undegraded":"2023-11-24T06:18:52.328117+0000","last_fullsized":"2023-11-24T06:18:52.328117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023821+0000","last_change":"2023-11-24T06:18:50.034218+0000","last_active":"2023-11-24T06:18:51.023821+0000","last_peered":"2023-11-24T06:18:51.023821+0000","last_clean":"2023-11-24T06:18:51.023821+0000","last_became_active":"2023-11-24T06:18:50.033754+0000","last_became_peered":"2023-11-24T06:18:50.033754+0000","last_unstale":"2023-11-24T06:18:51.023821+0000","last_undegraded":"2023-11-24T06:18:51.023821+0000","last_fullsized":"2023-11-24T06:18:51.023821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403228+0000","last_change":"2023-11-24T06:18:50.033709+0000","last_active":"2023-11-24T06:18:52.403228+0000","last_peered":"2023-11-24T06:18:52.403228+0000","last_clean":"2023-11-24T06:18:52.403228+0000","last_became_active":"2023-11-24T06:18:50.033447+0000","last_became_peered":"2023-11-24T06:18:50.033447+0000","last_unstale":"2023-11-24T06:18:52.403228+0000","last_undegraded":"2023-11-24T06:18:52.403228+0000","last_fullsized":"2023-11-24T06:18:52.403228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403761+0000","last_change":"2023-11-24T06:18:50.023134+0000","last_active":"2023-11-24T06:18:52.403761+0000","last_peered":"2023-11-24T06:18:52.403761+0000","last_clean":"2023-11-24T06:18:52.403761+0000","last_became_active":"2023-11-24T06:18:50.022837+0000","last_became_peered":"2023-11-24T06:18:50.022837+0000","last_unstale":"2023-11-24T06:18:52.403761+0000","last_undegraded":"2023-11-24T06:18:52.403761+0000","last_fullsized":"2023-11-24T06:18:52.403761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021102+0000","last_change":"2023-11-24T06:18:50.027089+0000","last_active":"2023-11-24T06:18:51.021102+0000","last_peered":"2023-11-24T06:18:51.021102+0000","last_clean":"2023-11-24T06:18:51.021102+0000","last_became_active":"2023-11-24T06:18:50.026810+0000","last_became_peered":"2023-11-24T06:18:50.026810+0000","last_unstale":"2023-11-24T06:18:51.021102+0000","last_undegraded":"2023-11-24T06:18:51.021102+0000","last_fullsized":"2023-11-24T06:18:51.021102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892218+0000","last_change":"2023-11-24T06:18:50.028542+0000","last_active":"2023-11-24T06:18:51.892218+0000","last_peered":"2023-11-24T06:18:51.892218+0000","last_clean":"2023-11-24T06:18:51.892218+0000","last_became_active":"2023-11-24T06:18:50.027971+0000","last_became_peered":"2023-11-24T06:18:50.027971+0000","last_unstale":"2023-11-24T06:18:51.892218+0000","last_undegraded":"2023-11-24T06:18:51.892218+0000","last_fullsized":"2023-11-24T06:18:51.892218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892968+0000","last_change":"2023-11-24T06:18:50.035313+0000","last_active":"2023-11-24T06:18:51.892968+0000","last_peered":"2023-11-24T06:18:51.892968+0000","last_clean":"2023-11-24T06:18:51.892968+0000","last_became_active":"2023-11-24T06:18:50.034744+0000","last_became_peered":"2023-11-24T06:18:50.034744+0000","last_unstale":"2023-11-24T06:18:51.892968+0000","last_undegraded":"2023-11-24T06:18:51.892968+0000","last_fullsized":"2023-11-24T06:18:51.892968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:03.577619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403701+0000","last_change":"2023-11-24T06:18:50.034072+0000","last_active":"2023-11-24T06:18:52.403701+0000","last_peered":"2023-11-24T06:18:52.403701+0000","last_clean":"2023-11-24T06:18:52.403701+0000","last_became_active":"2023-11-24T06:18:50.033798+0000","last_became_peered":"2023-11-24T06:18:50.033798+0000","last_unstale":"2023-11-24T06:18:52.403701+0000","last_undegraded":"2023-11-24T06:18:52.403701+0000","last_fullsized":"2023-11-24T06:18:52.403701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:14.231233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023796+0000","last_change":"2023-11-24T06:18:50.025458+0000","last_active":"2023-11-24T06:18:51.023796+0000","last_peered":"2023-11-24T06:18:51.023796+0000","last_clean":"2023-11-24T06:18:51.023796+0000","last_became_active":"2023-11-24T06:18:50.025139+0000","last_became_peered":"2023-11-24T06:18:50.025139+0000","last_unstale":"2023-11-24T06:18:51.023796+0000","last_undegraded":"2023-11-24T06:18:51.023796+0000","last_fullsized":"2023-11-24T06:18:51.023796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:02:29.652618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.735806+0000","last_change":"2023-11-24T06:18:49.009105+0000","last_active":"2023-11-24T06:18:51.735806+0000","last_peered":"2023-11-24T06:18:51.735806+0000","last_clean":"2023-11-24T06:18:51.735806+0000","last_became_active":"2023-11-24T06:18:49.008802+0000","last_became_peered":"2023-11-24T06:18:49.008802+0000","last_unstale":"2023-11-24T06:18:51.735806+0000","last_undegraded":"2023-11-24T06:18:51.735806+0000","last_fullsized":"2023-11-24T06:18:51.735806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403494+0000","last_change":"2023-11-24T06:18:50.033700+0000","last_active":"2023-11-24T06:18:52.403494+0000","last_peered":"2023-11-24T06:18:52.403494+0000","last_clean":"2023-11-24T06:18:52.403494+0000","last_became_active":"2023-11-24T06:18:50.033372+0000","last_became_peered":"2023-11-24T06:18:50.033372+0000","last_unstale":"2023-11-24T06:18:52.403494+0000","last_undegraded":"2023-11-24T06:18:52.403494+0000","last_fullsized":"2023-11-24T06:18:52.403494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:14.887594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.739276+0000","last_change":"2023-11-24T06:18:49.015649+0000","last_active":"2023-11-24T06:18:51.739276+0000","last_peered":"2023-11-24T06:18:51.739276+0000","last_clean":"2023-11-24T06:18:51.739276+0000","last_became_active":"2023-11-24T06:18:49.015329+0000","last_became_peered":"2023-11-24T06:18:49.015329+0000","last_unstale":"2023-11-24T06:18:51.739276+0000","last_undegraded":"2023-11-24T06:18:51.739276+0000","last_fullsized":"2023-11-24T06:18:51.739276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403874+0000","last_change":"2023-11-24T06:18:50.037859+0000","last_active":"2023-11-24T06:18:52.403874+0000","last_peered":"2023-11-24T06:18:52.403874+0000","last_clean":"2023-11-24T06:18:52.403874+0000","last_became_active":"2023-11-24T06:18:50.037565+0000","last_became_peered":"2023-11-24T06:18:50.037565+0000","last_unstale":"2023-11-24T06:18:52.403874+0000","last_undegraded":"2023-11-24T06:18:52.403874+0000","last_fullsized":"2023-11-24T06:18:52.403874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024207+0000","last_change":"2023-11-24T06:18:49.008317+0000","last_active":"2023-11-24T06:18:51.024207+0000","last_peered":"2023-11-24T06:18:51.024207+0000","last_clean":"2023-11-24T06:18:51.024207+0000","last_became_active":"2023-11-24T06:18:49.008039+0000","last_became_peered":"2023-11-24T06:18:49.008039+0000","last_unstale":"2023-11-24T06:18:51.024207+0000","last_undegraded":"2023-11-24T06:18:51.024207+0000","last_fullsized":"2023-11-24T06:18:51.024207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:58.320250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892514+0000","last_change":"2023-11-24T06:18:50.034882+0000","last_active":"2023-11-24T06:18:51.892514+0000","last_peered":"2023-11-24T06:18:51.892514+0000","last_clean":"2023-11-24T06:18:51.892514+0000","last_became_active":"2023-11-24T06:18:50.031623+0000","last_became_peered":"2023-11-24T06:18:50.031623+0000","last_unstale":"2023-11-24T06:18:51.892514+0000","last_undegraded":"2023-11-24T06:18:51.892514+0000","last_fullsized":"2023-11-24T06:18:51.892514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:14.887594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023720+0000","last_change":"2023-11-24T06:18:49.007346+0000","last_active":"2023-11-24T06:18:51.023720+0000","last_peered":"2023-11-24T06:18:51.023720+0000","last_clean":"2023-11-24T06:18:51.023720+0000","last_became_active":"2023-11-24T06:18:49.006300+0000","last_became_peered":"2023-11-24T06:18:49.006300+0000","last_unstale":"2023-11-24T06:18:51.023720+0000","last_undegraded":"2023-11-24T06:18:51.023720+0000","last_fullsized":"2023-11-24T06:18:51.023720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892742+0000","last_change":"2023-11-24T06:18:50.029883+0000","last_active":"2023-11-24T06:18:51.892742+0000","last_peered":"2023-11-24T06:18:51.892742+0000","last_clean":"2023-11-24T06:18:51.892742+0000","last_became_active":"2023-11-24T06:18:50.029618+0000","last_became_peered":"2023-11-24T06:18:50.029618+0000","last_unstale":"2023-11-24T06:18:51.892742+0000","last_undegraded":"2023-11-24T06:18:51.892742+0000","last_fullsized":"2023-11-24T06:18:51.892742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:32:20.935789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.739242+0000","last_change":"2023-11-24T06:18:49.015809+0000","last_active":"2023-11-24T06:18:51.739242+0000","last_peered":"2023-11-24T06:18:51.739242+0000","last_clean":"2023-11-24T06:18:51.739242+0000","last_became_active":"2023-11-24T06:18:49.015073+0000","last_became_peered":"2023-11-24T06:18:49.015073+0000","last_unstale":"2023-11-24T06:18:51.739242+0000","last_undegraded":"2023-11-24T06:18:51.739242+0000","last_fullsized":"2023-11-24T06:18:51.739242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:40.335012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023148+0000","last_change":"2023-11-24T06:18:50.015301+0000","last_active":"2023-11-24T06:18:51.023148+0000","last_peered":"2023-11-24T06:18:51.023148+0000","last_clean":"2023-11-24T06:18:51.023148+0000","last_became_active":"2023-11-24T06:18:50.015007+0000","last_became_peered":"2023-11-24T06:18:50.015007+0000","last_unstale":"2023-11-24T06:18:51.023148+0000","last_undegraded":"2023-11-24T06:18:51.023148+0000","last_fullsized":"2023-11-24T06:18:51.023148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:31.391676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738456+0000","last_change":"2023-11-24T06:18:49.008047+0000","last_active":"2023-11-24T06:18:51.738456+0000","last_peered":"2023-11-24T06:18:51.738456+0000","last_clean":"2023-11-24T06:18:51.738456+0000","last_became_active":"2023-11-24T06:18:49.007764+0000","last_became_peered":"2023-11-24T06:18:49.007764+0000","last_unstale":"2023-11-24T06:18:51.738456+0000","last_undegraded":"2023-11-24T06:18:51.738456+0000","last_fullsized":"2023-11-24T06:18:51.738456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:25:02.607697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403873+0000","last_change":"2023-11-24T06:18:50.027945+0000","last_active":"2023-11-24T06:18:52.403873+0000","last_peered":"2023-11-24T06:18:52.403873+0000","last_clean":"2023-11-24T06:18:52.403873+0000","last_became_active":"2023-11-24T06:18:50.027609+0000","last_became_peered":"2023-11-24T06:18:50.027609+0000","last_unstale":"2023-11-24T06:18:52.403873+0000","last_undegraded":"2023-11-24T06:18:52.403873+0000","last_fullsized":"2023-11-24T06:18:52.403873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.739190+0000","last_change":"2023-11-24T06:18:49.014611+0000","last_active":"2023-11-24T06:18:51.739190+0000","last_peered":"2023-11-24T06:18:51.739190+0000","last_clean":"2023-11-24T06:18:51.739190+0000","last_became_active":"2023-11-24T06:18:49.014277+0000","last_became_peered":"2023-11-24T06:18:49.014277+0000","last_unstale":"2023-11-24T06:18:51.739190+0000","last_undegraded":"2023-11-24T06:18:51.739190+0000","last_fullsized":"2023-11-24T06:18:51.739190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:33.478815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.402995+0000","last_change":"2023-11-24T06:18:50.027283+0000","last_active":"2023-11-24T06:18:52.402995+0000","last_peered":"2023-11-24T06:18:52.402995+0000","last_clean":"2023-11-24T06:18:52.402995+0000","last_became_active":"2023-11-24T06:18:50.027028+0000","last_became_peered":"2023-11-24T06:18:50.027028+0000","last_unstale":"2023-11-24T06:18:52.402995+0000","last_undegraded":"2023-11-24T06:18:52.402995+0000","last_fullsized":"2023-11-24T06:18:52.402995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738445+0000","last_change":"2023-11-24T06:18:49.007327+0000","last_active":"2023-11-24T06:18:51.738445+0000","last_peered":"2023-11-24T06:18:51.738445+0000","last_clean":"2023-11-24T06:18:51.738445+0000","last_became_active":"2023-11-24T06:18:49.006684+0000","last_became_peered":"2023-11-24T06:18:49.006684+0000","last_unstale":"2023-11-24T06:18:51.738445+0000","last_undegraded":"2023-11-24T06:18:51.738445+0000","last_fullsized":"2023-11-24T06:18:51.738445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:54.335883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021189+0000","last_change":"2023-11-24T06:18:50.026294+0000","last_active":"2023-11-24T06:18:51.021189+0000","last_peered":"2023-11-24T06:18:51.021189+0000","last_clean":"2023-11-24T06:18:51.021189+0000","last_became_active":"2023-11-24T06:18:50.026020+0000","last_became_peered":"2023-11-24T06:18:50.026020+0000","last_unstale":"2023-11-24T06:18:51.021189+0000","last_undegraded":"2023-11-24T06:18:51.021189+0000","last_fullsized":"2023-11-24T06:18:51.021189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738579+0000","last_change":"2023-11-24T06:18:49.011149+0000","last_active":"2023-11-24T06:18:51.738579+0000","last_peered":"2023-11-24T06:18:51.738579+0000","last_clean":"2023-11-24T06:18:51.738579+0000","last_became_active":"2023-11-24T06:18:49.010573+0000","last_became_peered":"2023-11-24T06:18:49.010573+0000","last_unstale":"2023-11-24T06:18:51.738579+0000","last_undegraded":"2023-11-24T06:18:51.738579+0000","last_fullsized":"2023-11-24T06:18:51.738579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892835+0000","last_change":"2023-11-24T06:18:50.035496+0000","last_active":"2023-11-24T06:18:51.892835+0000","last_peered":"2023-11-24T06:18:51.892835+0000","last_clean":"2023-11-24T06:18:51.892835+0000","last_became_active":"2023-11-24T06:18:50.035172+0000","last_became_peered":"2023-11-24T06:18:50.035172+0000","last_unstale":"2023-11-24T06:18:51.892835+0000","last_undegraded":"2023-11-24T06:18:51.892835+0000","last_fullsized":"2023-11-24T06:18:51.892835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:12:39.141407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738707+0000","last_change":"2023-11-24T06:18:49.012031+0000","last_active":"2023-11-24T06:18:51.738707+0000","last_peered":"2023-11-24T06:18:51.738707+0000","last_clean":"2023-11-24T06:18:51.738707+0000","last_became_active":"2023-11-24T06:18:49.011741+0000","last_became_peered":"2023-11-24T06:18:49.011741+0000","last_unstale":"2023-11-24T06:18:51.738707+0000","last_undegraded":"2023-11-24T06:18:51.738707+0000","last_fullsized":"2023-11-24T06:18:51.738707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.404093+0000","last_change":"2023-11-24T06:18:50.035543+0000","last_active":"2023-11-24T06:18:52.404093+0000","last_peered":"2023-11-24T06:18:52.404093+0000","last_clean":"2023-11-24T06:18:52.404093+0000","last_became_active":"2023-11-24T06:18:50.035269+0000","last_became_peered":"2023-11-24T06:18:50.035269+0000","last_unstale":"2023-11-24T06:18:52.404093+0000","last_undegraded":"2023-11-24T06:18:52.404093+0000","last_fullsized":"2023-11-24T06:18:52.404093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021480+0000","last_change":"2023-11-24T06:18:49.006503+0000","last_active":"2023-11-24T06:18:51.021480+0000","last_peered":"2023-11-24T06:18:51.021480+0000","last_clean":"2023-11-24T06:18:51.021480+0000","last_became_active":"2023-11-24T06:18:49.006176+0000","last_became_peered":"2023-11-24T06:18:49.006176+0000","last_unstale":"2023-11-24T06:18:51.021480+0000","last_undegraded":"2023-11-24T06:18:51.021480+0000","last_fullsized":"2023-11-24T06:18:51.021480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:33.478815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023461+0000","last_change":"2023-11-24T06:18:50.023420+0000","last_active":"2023-11-24T06:18:51.023461+0000","last_peered":"2023-11-24T06:18:51.023461+0000","last_clean":"2023-11-24T06:18:51.023461+0000","last_became_active":"2023-11-24T06:18:50.022722+0000","last_became_peered":"2023-11-24T06:18:50.022722+0000","last_unstale":"2023-11-24T06:18:51.023461+0000","last_undegraded":"2023-11-24T06:18:51.023461+0000","last_fullsized":"2023-11-24T06:18:51.023461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:24.709998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738477+0000","last_change":"2023-11-24T06:18:49.016525+0000","last_active":"2023-11-24T06:18:51.738477+0000","last_peered":"2023-11-24T06:18:51.738477+0000","last_clean":"2023-11-24T06:18:51.738477+0000","last_became_active":"2023-11-24T06:18:49.016259+0000","last_became_peered":"2023-11-24T06:18:49.016259+0000","last_unstale":"2023-11-24T06:18:51.738477+0000","last_undegraded":"2023-11-24T06:18:51.738477+0000","last_fullsized":"2023-11-24T06:18:51.738477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:36:13.224297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023938+0000","last_change":"2023-11-24T06:18:50.024377+0000","last_active":"2023-11-24T06:18:51.023938+0000","last_peered":"2023-11-24T06:18:51.023938+0000","last_clean":"2023-11-24T06:18:51.023938+0000","last_became_active":"2023-11-24T06:18:50.024123+0000","last_became_peered":"2023-11-24T06:18:50.024123+0000","last_unstale":"2023-11-24T06:18:51.023938+0000","last_undegraded":"2023-11-24T06:18:51.023938+0000","last_fullsized":"2023-11-24T06:18:51.023938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:28:39.352489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.735753+0000","last_change":"2023-11-24T06:18:49.009176+0000","last_active":"2023-11-24T06:18:51.735753+0000","last_peered":"2023-11-24T06:18:51.735753+0000","last_clean":"2023-11-24T06:18:51.735753+0000","last_became_active":"2023-11-24T06:18:49.008875+0000","last_became_peered":"2023-11-24T06:18:49.008875+0000","last_unstale":"2023-11-24T06:18:51.735753+0000","last_undegraded":"2023-11-24T06:18:51.735753+0000","last_fullsized":"2023-11-24T06:18:51.735753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403625+0000","last_change":"2023-11-24T06:18:50.032393+0000","last_active":"2023-11-24T06:18:52.403625+0000","last_peered":"2023-11-24T06:18:52.403625+0000","last_clean":"2023-11-24T06:18:52.403625+0000","last_became_active":"2023-11-24T06:18:50.032145+0000","last_became_peered":"2023-11-24T06:18:50.032145+0000","last_unstale":"2023-11-24T06:18:52.403625+0000","last_undegraded":"2023-11-24T06:18:52.403625+0000","last_fullsized":"2023-11-24T06:18:52.403625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:23:34.485750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738587+0000","last_change":"2023-11-24T06:18:49.015579+0000","last_active":"2023-11-24T06:18:51.738587+0000","last_peered":"2023-11-24T06:18:51.738587+0000","last_clean":"2023-11-24T06:18:51.738587+0000","last_became_active":"2023-11-24T06:18:49.015329+0000","last_became_peered":"2023-11-24T06:18:49.015329+0000","last_unstale":"2023-11-24T06:18:51.738587+0000","last_undegraded":"2023-11-24T06:18:51.738587+0000","last_fullsized":"2023-11-24T06:18:51.738587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:37.250753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024128+0000","last_change":"2023-11-24T06:18:50.021964+0000","last_active":"2023-11-24T06:18:51.024128+0000","last_peered":"2023-11-24T06:18:51.024128+0000","last_clean":"2023-11-24T06:18:51.024128+0000","last_became_active":"2023-11-24T06:18:50.021678+0000","last_became_peered":"2023-11-24T06:18:50.021678+0000","last_unstale":"2023-11-24T06:18:51.024128+0000","last_undegraded":"2023-11-24T06:18:51.024128+0000","last_fullsized":"2023-11-24T06:18:51.024128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:32:20.935789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738432+0000","last_change":"2023-11-24T06:18:49.007859+0000","last_active":"2023-11-24T06:18:51.738432+0000","last_peered":"2023-11-24T06:18:51.738432+0000","last_clean":"2023-11-24T06:18:51.738432+0000","last_became_active":"2023-11-24T06:18:49.007569+0000","last_became_peered":"2023-11-24T06:18:49.007569+0000","last_unstale":"2023-11-24T06:18:51.738432+0000","last_undegraded":"2023-11-24T06:18:51.738432+0000","last_fullsized":"2023-11-24T06:18:51.738432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.327901+0000","last_change":"2023-11-24T06:18:50.033953+0000","last_active":"2023-11-24T06:18:52.327901+0000","last_peered":"2023-11-24T06:18:52.327901+0000","last_clean":"2023-11-24T06:18:52.327901+0000","last_became_active":"2023-11-24T06:18:50.033664+0000","last_became_peered":"2023-11-24T06:18:50.033664+0000","last_unstale":"2023-11-24T06:18:52.327901+0000","last_undegraded":"2023-11-24T06:18:52.327901+0000","last_fullsized":"2023-11-24T06:18:52.327901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:55.342818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745781+0000","last_change":"2023-11-24T06:18:49.009433+0000","last_active":"2023-11-24T06:18:51.745781+0000","last_peered":"2023-11-24T06:18:51.745781+0000","last_clean":"2023-11-24T06:18:51.745781+0000","last_became_active":"2023-11-24T06:18:49.009133+0000","last_became_peered":"2023-11-24T06:18:49.009133+0000","last_unstale":"2023-11-24T06:18:51.745781+0000","last_undegraded":"2023-11-24T06:18:51.745781+0000","last_fullsized":"2023-11-24T06:18:51.745781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023055+0000","last_change":"2023-11-24T06:18:50.023048+0000","last_active":"2023-11-24T06:18:51.023055+0000","last_peered":"2023-11-24T06:18:51.023055+0000","last_clean":"2023-11-24T06:18:51.023055+0000","last_became_active":"2023-11-24T06:18:50.022736+0000","last_became_peered":"2023-11-24T06:18:50.022736+0000","last_unstale":"2023-11-24T06:18:51.023055+0000","last_undegraded":"2023-11-24T06:18:51.023055+0000","last_fullsized":"2023-11-24T06:18:51.023055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:46:14.887594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023699+0000","last_change":"2023-11-24T06:18:49.012343+0000","last_active":"2023-11-24T06:18:51.023699+0000","last_peered":"2023-11-24T06:18:51.023699+0000","last_clean":"2023-11-24T06:18:51.023699+0000","last_became_active":"2023-11-24T06:18:49.012078+0000","last_became_peered":"2023-11-24T06:18:49.012078+0000","last_unstale":"2023-11-24T06:18:51.023699+0000","last_undegraded":"2023-11-24T06:18:51.023699+0000","last_fullsized":"2023-11-24T06:18:51.023699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:40:26.842643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024206+0000","last_change":"2023-11-24T06:18:50.022294+0000","last_active":"2023-11-24T06:18:51.024206+0000","last_peered":"2023-11-24T06:18:51.024206+0000","last_clean":"2023-11-24T06:18:51.024206+0000","last_became_active":"2023-11-24T06:18:50.021806+0000","last_became_peered":"2023-11-24T06:18:50.021806+0000","last_unstale":"2023-11-24T06:18:51.024206+0000","last_undegraded":"2023-11-24T06:18:51.024206+0000","last_fullsized":"2023-11-24T06:18:51.024206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:03.577619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021419+0000","last_change":"2023-11-24T06:18:49.010001+0000","last_active":"2023-11-24T06:18:51.021419+0000","last_peered":"2023-11-24T06:18:51.021419+0000","last_clean":"2023-11-24T06:18:51.021419+0000","last_became_active":"2023-11-24T06:18:49.009729+0000","last_became_peered":"2023-11-24T06:18:49.009729+0000","last_unstale":"2023-11-24T06:18:51.021419+0000","last_undegraded":"2023-11-24T06:18:51.021419+0000","last_fullsized":"2023-11-24T06:18:51.021419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:24.571482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892753+0000","last_change":"2023-11-24T06:18:50.035809+0000","last_active":"2023-11-24T06:18:51.892753+0000","last_peered":"2023-11-24T06:18:51.892753+0000","last_clean":"2023-11-24T06:18:51.892753+0000","last_became_active":"2023-11-24T06:18:50.033211+0000","last_became_peered":"2023-11-24T06:18:50.033211+0000","last_unstale":"2023-11-24T06:18:51.892753+0000","last_undegraded":"2023-11-24T06:18:51.892753+0000","last_fullsized":"2023-11-24T06:18:51.892753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:33:00.547188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738562+0000","last_change":"2023-11-24T06:18:49.004923+0000","last_active":"2023-11-24T06:18:51.738562+0000","last_peered":"2023-11-24T06:18:51.738562+0000","last_clean":"2023-11-24T06:18:51.738562+0000","last_became_active":"2023-11-24T06:18:49.004601+0000","last_became_peered":"2023-11-24T06:18:49.004601+0000","last_unstale":"2023-11-24T06:18:51.738562+0000","last_undegraded":"2023-11-24T06:18:51.738562+0000","last_fullsized":"2023-11-24T06:18:51.738562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:42.514481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023365+0000","last_change":"2023-11-24T06:18:50.026574+0000","last_active":"2023-11-24T06:18:51.023365+0000","last_peered":"2023-11-24T06:18:51.023365+0000","last_clean":"2023-11-24T06:18:51.023365+0000","last_became_active":"2023-11-24T06:18:50.026028+0000","last_became_peered":"2023-11-24T06:18:50.026028+0000","last_unstale":"2023-11-24T06:18:51.023365+0000","last_undegraded":"2023-11-24T06:18:51.023365+0000","last_fullsized":"2023-11-24T06:18:51.023365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:02.174822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021109+0000","last_change":"2023-11-24T06:18:50.027544+0000","last_active":"2023-11-24T06:18:51.021109+0000","last_peered":"2023-11-24T06:18:51.021109+0000","last_clean":"2023-11-24T06:18:51.021109+0000","last_became_active":"2023-11-24T06:18:50.027272+0000","last_became_peered":"2023-11-24T06:18:50.027272+0000","last_unstale":"2023-11-24T06:18:51.021109+0000","last_undegraded":"2023-11-24T06:18:51.021109+0000","last_fullsized":"2023-11-24T06:18:51.021109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745828+0000","last_change":"2023-11-24T06:18:49.015756+0000","last_active":"2023-11-24T06:18:51.745828+0000","last_peered":"2023-11-24T06:18:51.745828+0000","last_clean":"2023-11-24T06:18:51.745828+0000","last_became_active":"2023-11-24T06:18:49.015446+0000","last_became_peered":"2023-11-24T06:18:49.015446+0000","last_unstale":"2023-11-24T06:18:51.745828+0000","last_undegraded":"2023-11-24T06:18:51.745828+0000","last_fullsized":"2023-11-24T06:18:51.745828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:48:24.571482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403228+0000","last_change":"2023-11-24T06:18:50.040018+0000","last_active":"2023-11-24T06:18:52.403228+0000","last_peered":"2023-11-24T06:18:52.403228+0000","last_clean":"2023-11-24T06:18:52.403228+0000","last_became_active":"2023-11-24T06:18:50.039786+0000","last_became_peered":"2023-11-24T06:18:50.039786+0000","last_unstale":"2023-11-24T06:18:52.403228+0000","last_undegraded":"2023-11-24T06:18:52.403228+0000","last_fullsized":"2023-11-24T06:18:52.403228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738619+0000","last_change":"2023-11-24T06:18:49.009457+0000","last_active":"2023-11-24T06:18:51.738619+0000","last_peered":"2023-11-24T06:18:51.738619+0000","last_clean":"2023-11-24T06:18:51.738619+0000","last_became_active":"2023-11-24T06:18:49.008932+0000","last_became_peered":"2023-11-24T06:18:49.008932+0000","last_unstale":"2023-11-24T06:18:51.738619+0000","last_undegraded":"2023-11-24T06:18:51.738619+0000","last_fullsized":"2023-11-24T06:18:51.738619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:23.703063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:50.009997+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:18:50.009997+0000","last_peered":"2023-11-24T06:18:50.009997+0000","last_clean":"2023-11-24T06:18:50.009997+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:18:50.009997+0000","last_undegraded":"2023-11-24T06:18:50.009997+0000","last_fullsized":"2023-11-24T06:18:50.009997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403625+0000","last_change":"2023-11-24T06:18:50.027632+0000","last_active":"2023-11-24T06:18:52.403625+0000","last_peered":"2023-11-24T06:18:52.403625+0000","last_clean":"2023-11-24T06:18:52.403625+0000","last_became_active":"2023-11-24T06:18:50.027344+0000","last_became_peered":"2023-11-24T06:18:50.027344+0000","last_unstale":"2023-11-24T06:18:52.403625+0000","last_undegraded":"2023-11-24T06:18:52.403625+0000","last_fullsized":"2023-11-24T06:18:52.403625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:56:59.481984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022810+0000","last_change":"2023-11-24T06:18:49.006996+0000","last_active":"2023-11-24T06:18:51.022810+0000","last_peered":"2023-11-24T06:18:51.022810+0000","last_clean":"2023-11-24T06:18:51.022810+0000","last_became_active":"2023-11-24T06:18:49.006652+0000","last_became_peered":"2023-11-24T06:18:49.006652+0000","last_unstale":"2023-11-24T06:18:51.022810+0000","last_undegraded":"2023-11-24T06:18:51.022810+0000","last_fullsized":"2023-11-24T06:18:51.022810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:01.167887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024055+0000","last_change":"2023-11-24T06:18:49.012066+0000","last_active":"2023-11-24T06:18:51.024055+0000","last_peered":"2023-11-24T06:18:51.024055+0000","last_clean":"2023-11-24T06:18:51.024055+0000","last_became_active":"2023-11-24T06:18:49.011795+0000","last_became_peered":"2023-11-24T06:18:49.011795+0000","last_unstale":"2023-11-24T06:18:51.024055+0000","last_undegraded":"2023-11-24T06:18:51.024055+0000","last_fullsized":"2023-11-24T06:18:51.024055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:22.850843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021314+0000","last_change":"2023-11-24T06:18:50.021739+0000","last_active":"2023-11-24T06:18:51.021314+0000","last_peered":"2023-11-24T06:18:51.021314+0000","last_clean":"2023-11-24T06:18:51.021314+0000","last_became_active":"2023-11-24T06:18:50.021452+0000","last_became_peered":"2023-11-24T06:18:50.021452+0000","last_unstale":"2023-11-24T06:18:51.021314+0000","last_undegraded":"2023-11-24T06:18:51.021314+0000","last_fullsized":"2023-11-24T06:18:51.021314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745909+0000","last_change":"2023-11-24T06:18:49.012066+0000","last_active":"2023-11-24T06:18:51.745909+0000","last_peered":"2023-11-24T06:18:51.745909+0000","last_clean":"2023-11-24T06:18:51.745909+0000","last_became_active":"2023-11-24T06:18:49.011748+0000","last_became_peered":"2023-11-24T06:18:49.011748+0000","last_unstale":"2023-11-24T06:18:51.745909+0000","last_undegraded":"2023-11-24T06:18:51.745909+0000","last_fullsized":"2023-11-24T06:18:51.745909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.404100+0000","last_change":"2023-11-24T06:18:50.033206+0000","last_active":"2023-11-24T06:18:52.404100+0000","last_peered":"2023-11-24T06:18:52.404100+0000","last_clean":"2023-11-24T06:18:52.404100+0000","last_became_active":"2023-11-24T06:18:50.032878+0000","last_became_peered":"2023-11-24T06:18:50.032878+0000","last_unstale":"2023-11-24T06:18:52.404100+0000","last_undegraded":"2023-11-24T06:18:52.404100+0000","last_fullsized":"2023-11-24T06:18:52.404100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:28:39.352489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.738568+0000","last_change":"2023-11-24T06:18:49.011356+0000","last_active":"2023-11-24T06:18:51.738568+0000","last_peered":"2023-11-24T06:18:51.738568+0000","last_clean":"2023-11-24T06:18:51.738568+0000","last_became_active":"2023-11-24T06:18:49.010903+0000","last_became_peered":"2023-11-24T06:18:49.010903+0000","last_unstale":"2023-11-24T06:18:51.738568+0000","last_undegraded":"2023-11-24T06:18:51.738568+0000","last_fullsized":"2023-11-24T06:18:51.738568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403395+0000","last_change":"2023-11-24T06:18:50.024110+0000","last_active":"2023-11-24T06:18:52.403395+0000","last_peered":"2023-11-24T06:18:52.403395+0000","last_clean":"2023-11-24T06:18:52.403395+0000","last_became_active":"2023-11-24T06:18:50.023801+0000","last_became_peered":"2023-11-24T06:18:50.023801+0000","last_unstale":"2023-11-24T06:18:52.403395+0000","last_undegraded":"2023-11-24T06:18:52.403395+0000","last_fullsized":"2023-11-24T06:18:52.403395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:54.282518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.022707+0000","last_change":"2023-11-24T06:18:49.010428+0000","last_active":"2023-11-24T06:18:51.022707+0000","last_peered":"2023-11-24T06:18:51.022707+0000","last_clean":"2023-11-24T06:18:51.022707+0000","last_became_active":"2023-11-24T06:18:49.010124+0000","last_became_peered":"2023-11-24T06:18:49.010124+0000","last_unstale":"2023-11-24T06:18:51.022707+0000","last_undegraded":"2023-11-24T06:18:51.022707+0000","last_fullsized":"2023-11-24T06:18:51.022707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.023378+0000","last_change":"2023-11-24T06:18:50.022092+0000","last_active":"2023-11-24T06:18:51.023378+0000","last_peered":"2023-11-24T06:18:51.023378+0000","last_clean":"2023-11-24T06:18:51.023378+0000","last_became_active":"2023-11-24T06:18:50.021808+0000","last_became_peered":"2023-11-24T06:18:50.021808+0000","last_unstale":"2023-11-24T06:18:51.023378+0000","last_undegraded":"2023-11-24T06:18:51.023378+0000","last_fullsized":"2023-11-24T06:18:51.023378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:21.027784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021058+0000","last_change":"2023-11-24T06:18:49.007474+0000","last_active":"2023-11-24T06:18:51.021058+0000","last_peered":"2023-11-24T06:18:51.021058+0000","last_clean":"2023-11-24T06:18:51.021058+0000","last_became_active":"2023-11-24T06:18:49.007228+0000","last_became_peered":"2023-11-24T06:18:49.007228+0000","last_unstale":"2023-11-24T06:18:51.021058+0000","last_undegraded":"2023-11-24T06:18:51.021058+0000","last_fullsized":"2023-11-24T06:18:51.021058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403034+0000","last_change":"2023-11-24T06:18:50.025141+0000","last_active":"2023-11-24T06:18:52.403034+0000","last_peered":"2023-11-24T06:18:52.403034+0000","last_clean":"2023-11-24T06:18:52.403034+0000","last_became_active":"2023-11-24T06:18:50.024859+0000","last_became_peered":"2023-11-24T06:18:50.024859+0000","last_unstale":"2023-11-24T06:18:52.403034+0000","last_undegraded":"2023-11-24T06:18:52.403034+0000","last_fullsized":"2023-11-24T06:18:52.403034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021233+0000","last_change":"2023-11-24T06:18:49.015615+0000","last_active":"2023-11-24T06:18:51.021233+0000","last_peered":"2023-11-24T06:18:51.021233+0000","last_clean":"2023-11-24T06:18:51.021233+0000","last_became_active":"2023-11-24T06:18:49.015320+0000","last_became_peered":"2023-11-24T06:18:49.015320+0000","last_unstale":"2023-11-24T06:18:51.021233+0000","last_undegraded":"2023-11-24T06:18:51.021233+0000","last_fullsized":"2023-11-24T06:18:51.021233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:54:20.020849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403348+0000","last_change":"2023-11-24T06:18:50.025429+0000","last_active":"2023-11-24T06:18:52.403348+0000","last_peered":"2023-11-24T06:18:52.403348+0000","last_clean":"2023-11-24T06:18:52.403348+0000","last_became_active":"2023-11-24T06:18:50.024972+0000","last_became_peered":"2023-11-24T06:18:50.024972+0000","last_unstale":"2023-11-24T06:18:52.403348+0000","last_undegraded":"2023-11-24T06:18:52.403348+0000","last_fullsized":"2023-11-24T06:18:52.403348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:48.527966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024261+0000","last_change":"2023-11-24T06:18:49.007100+0000","last_active":"2023-11-24T06:18:51.024261+0000","last_peered":"2023-11-24T06:18:51.024261+0000","last_clean":"2023-11-24T06:18:51.024261+0000","last_became_active":"2023-11-24T06:18:49.006577+0000","last_became_peered":"2023-11-24T06:18:49.006577+0000","last_unstale":"2023-11-24T06:18:51.024261+0000","last_undegraded":"2023-11-24T06:18:51.024261+0000","last_fullsized":"2023-11-24T06:18:51.024261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:53.275582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328023+0000","last_change":"2023-11-24T06:18:50.032437+0000","last_active":"2023-11-24T06:18:52.328023+0000","last_peered":"2023-11-24T06:18:52.328023+0000","last_clean":"2023-11-24T06:18:52.328023+0000","last_became_active":"2023-11-24T06:18:50.032082+0000","last_became_peered":"2023-11-24T06:18:50.032082+0000","last_unstale":"2023-11-24T06:18:52.328023+0000","last_undegraded":"2023-11-24T06:18:52.328023+0000","last_fullsized":"2023-11-24T06:18:52.328023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:38.257688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745848+0000","last_change":"2023-11-24T06:18:49.011290+0000","last_active":"2023-11-24T06:18:51.745848+0000","last_peered":"2023-11-24T06:18:51.745848+0000","last_clean":"2023-11-24T06:18:51.745848+0000","last_became_active":"2023-11-24T06:18:49.010889+0000","last_became_peered":"2023-11-24T06:18:49.010889+0000","last_unstale":"2023-11-24T06:18:51.745848+0000","last_undegraded":"2023-11-24T06:18:51.745848+0000","last_fullsized":"2023-11-24T06:18:51.745848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:40:30.384741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024098+0000","last_change":"2023-11-24T06:18:50.036167+0000","last_active":"2023-11-24T06:18:51.024098+0000","last_peered":"2023-11-24T06:18:51.024098+0000","last_clean":"2023-11-24T06:18:51.024098+0000","last_became_active":"2023-11-24T06:18:50.035904+0000","last_became_peered":"2023-11-24T06:18:50.035904+0000","last_unstale":"2023-11-24T06:18:51.024098+0000","last_undegraded":"2023-11-24T06:18:51.024098+0000","last_fullsized":"2023-11-24T06:18:51.024098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:33:00.547188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021448+0000","last_change":"2023-11-24T06:18:49.006925+0000","last_active":"2023-11-24T06:18:51.021448+0000","last_peered":"2023-11-24T06:18:51.021448+0000","last_clean":"2023-11-24T06:18:51.021448+0000","last_became_active":"2023-11-24T06:18:49.006482+0000","last_became_peered":"2023-11-24T06:18:49.006482+0000","last_unstale":"2023-11-24T06:18:51.021448+0000","last_undegraded":"2023-11-24T06:18:51.021448+0000","last_fullsized":"2023-11-24T06:18:51.021448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:40.335012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892543+0000","last_change":"2023-11-24T06:18:50.039417+0000","last_active":"2023-11-24T06:18:51.892543+0000","last_peered":"2023-11-24T06:18:51.892543+0000","last_clean":"2023-11-24T06:18:51.892543+0000","last_became_active":"2023-11-24T06:18:50.031461+0000","last_became_peered":"2023-11-24T06:18:50.031461+0000","last_unstale":"2023-11-24T06:18:51.892543+0000","last_undegraded":"2023-11-24T06:18:51.892543+0000","last_fullsized":"2023-11-24T06:18:51.892543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:28:39.352489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.745887+0000","last_change":"2023-11-24T06:18:49.015786+0000","last_active":"2023-11-24T06:18:51.745887+0000","last_peered":"2023-11-24T06:18:51.745887+0000","last_clean":"2023-11-24T06:18:51.745887+0000","last_became_active":"2023-11-24T06:18:49.015506+0000","last_became_peered":"2023-11-24T06:18:49.015506+0000","last_unstale":"2023-11-24T06:18:51.745887+0000","last_undegraded":"2023-11-24T06:18:51.745887+0000","last_fullsized":"2023-11-24T06:18:51.745887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:47.980431+0000","last_clean_scrub_stamp":"2023-11-24T06:18:47.980431+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:18:01.167887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.024358+0000","last_change":"2023-11-24T06:18:50.023402+0000","last_active":"2023-11-24T06:18:51.024358+0000","last_peered":"2023-11-24T06:18:51.024358+0000","last_clean":"2023-11-24T06:18:51.024358+0000","last_became_active":"2023-11-24T06:18:50.023117+0000","last_became_peered":"2023-11-24T06:18:50.023117+0000","last_unstale":"2023-11-24T06:18:51.024358+0000","last_undegraded":"2023-11-24T06:18:51.024358+0000","last_fullsized":"2023-11-24T06:18:51.024358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:54.022789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.327886+0000","last_change":"2023-11-24T06:18:50.032212+0000","last_active":"2023-11-24T06:18:52.327886+0000","last_peered":"2023-11-24T06:18:52.327886+0000","last_clean":"2023-11-24T06:18:52.327886+0000","last_became_active":"2023-11-24T06:18:50.031933+0000","last_became_peered":"2023-11-24T06:18:50.031933+0000","last_unstale":"2023-11-24T06:18:52.327886+0000","last_undegraded":"2023-11-24T06:18:52.327886+0000","last_fullsized":"2023-11-24T06:18:52.327886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:56:59.481984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403264+0000","last_change":"2023-11-24T06:18:50.037725+0000","last_active":"2023-11-24T06:18:52.403264+0000","last_peered":"2023-11-24T06:18:52.403264+0000","last_clean":"2023-11-24T06:18:52.403264+0000","last_became_active":"2023-11-24T06:18:50.037471+0000","last_became_peered":"2023-11-24T06:18:50.037471+0000","last_unstale":"2023-11-24T06:18:52.403264+0000","last_undegraded":"2023-11-24T06:18:52.403264+0000","last_fullsized":"2023-11-24T06:18:52.403264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:33:24.709998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892512+0000","last_change":"2023-11-24T06:18:50.038973+0000","last_active":"2023-11-24T06:18:51.892512+0000","last_peered":"2023-11-24T06:18:51.892512+0000","last_clean":"2023-11-24T06:18:51.892512+0000","last_became_active":"2023-11-24T06:18:50.038345+0000","last_became_peered":"2023-11-24T06:18:50.038345+0000","last_unstale":"2023-11-24T06:18:51.892512+0000","last_undegraded":"2023-11-24T06:18:51.892512+0000","last_fullsized":"2023-11-24T06:18:51.892512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:02:29.652618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403787+0000","last_change":"2023-11-24T06:18:50.024376+0000","last_active":"2023-11-24T06:18:52.403787+0000","last_peered":"2023-11-24T06:18:52.403787+0000","last_clean":"2023-11-24T06:18:52.403787+0000","last_became_active":"2023-11-24T06:18:50.024118+0000","last_became_peered":"2023-11-24T06:18:50.024118+0000","last_unstale":"2023-11-24T06:18:52.403787+0000","last_undegraded":"2023-11-24T06:18:52.403787+0000","last_fullsized":"2023-11-24T06:18:52.403787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:42:43.521416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403334+0000","last_change":"2023-11-24T06:18:50.028334+0000","last_active":"2023-11-24T06:18:52.403334+0000","last_peered":"2023-11-24T06:18:52.403334+0000","last_clean":"2023-11-24T06:18:52.403334+0000","last_became_active":"2023-11-24T06:18:50.028091+0000","last_became_peered":"2023-11-24T06:18:50.028091+0000","last_unstale":"2023-11-24T06:18:52.403334+0000","last_undegraded":"2023-11-24T06:18:52.403334+0000","last_fullsized":"2023-11-24T06:18:52.403334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:59.327185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.021380+0000","last_change":"2023-11-24T06:18:50.025563+0000","last_active":"2023-11-24T06:18:51.021380+0000","last_peered":"2023-11-24T06:18:51.021380+0000","last_clean":"2023-11-24T06:18:51.021380+0000","last_became_active":"2023-11-24T06:18:50.025276+0000","last_became_peered":"2023-11-24T06:18:50.025276+0000","last_unstale":"2023-11-24T06:18:51.021380+0000","last_undegraded":"2023-11-24T06:18:51.021380+0000","last_fullsized":"2023-11-24T06:18:51.021380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:17.715824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328320+0000","last_change":"2023-11-24T06:18:50.032672+0000","last_active":"2023-11-24T06:18:52.328320+0000","last_peered":"2023-11-24T06:18:52.328320+0000","last_clean":"2023-11-24T06:18:52.328320+0000","last_became_active":"2023-11-24T06:18:50.032400+0000","last_became_peered":"2023-11-24T06:18:50.032400+0000","last_unstale":"2023-11-24T06:18:52.328320+0000","last_undegraded":"2023-11-24T06:18:52.328320+0000","last_fullsized":"2023-11-24T06:18:52.328320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:23.857779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.404010+0000","last_change":"2023-11-24T06:18:50.027189+0000","last_active":"2023-11-24T06:18:52.404010+0000","last_peered":"2023-11-24T06:18:52.404010+0000","last_clean":"2023-11-24T06:18:52.404010+0000","last_became_active":"2023-11-24T06:18:50.026893+0000","last_became_peered":"2023-11-24T06:18:50.026893+0000","last_unstale":"2023-11-24T06:18:52.404010+0000","last_undegraded":"2023-11-24T06:18:52.404010+0000","last_fullsized":"2023-11-24T06:18:52.404010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:02:29.652618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892394+0000","last_change":"2023-11-24T06:18:50.031274+0000","last_active":"2023-11-24T06:18:51.892394+0000","last_peered":"2023-11-24T06:18:51.892394+0000","last_clean":"2023-11-24T06:18:51.892394+0000","last_became_active":"2023-11-24T06:18:50.030997+0000","last_became_peered":"2023-11-24T06:18:50.030997+0000","last_unstale":"2023-11-24T06:18:51.892394+0000","last_undegraded":"2023-11-24T06:18:51.892394+0000","last_fullsized":"2023-11-24T06:18:51.892394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:59.327185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403094+0000","last_change":"2023-11-24T06:18:50.027675+0000","last_active":"2023-11-24T06:18:52.403094+0000","last_peered":"2023-11-24T06:18:52.403094+0000","last_clean":"2023-11-24T06:18:52.403094+0000","last_became_active":"2023-11-24T06:18:50.027402+0000","last_became_peered":"2023-11-24T06:18:50.027402+0000","last_unstale":"2023-11-24T06:18:52.403094+0000","last_undegraded":"2023-11-24T06:18:52.403094+0000","last_fullsized":"2023-11-24T06:18:52.403094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:51.892381+0000","last_change":"2023-11-24T06:18:50.039174+0000","last_active":"2023-11-24T06:18:51.892381+0000","last_peered":"2023-11-24T06:18:51.892381+0000","last_clean":"2023-11-24T06:18:51.892381+0000","last_became_active":"2023-11-24T06:18:50.038671+0000","last_became_peered":"2023-11-24T06:18:50.038671+0000","last_unstale":"2023-11-24T06:18:51.892381+0000","last_undegraded":"2023-11-24T06:18:51.892381+0000","last_fullsized":"2023-11-24T06:18:51.892381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:56:48.527966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.403711+0000","last_change":"2023-11-24T06:18:50.032515+0000","last_active":"2023-11-24T06:18:52.403711+0000","last_peered":"2023-11-24T06:18:52.403711+0000","last_clean":"2023-11-24T06:18:52.403711+0000","last_became_active":"2023-11-24T06:18:50.032238+0000","last_became_peered":"2023-11-24T06:18:50.032238+0000","last_unstale":"2023-11-24T06:18:52.403711+0000","last_undegraded":"2023-11-24T06:18:52.403711+0000","last_fullsized":"2023-11-24T06:18:52.403711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:19:55.342818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-24T06:18:52.328180+0000","last_change":"2023-11-24T06:18:50.035460+0000","last_active":"2023-11-24T06:18:52.328180+0000","last_peered":"2023-11-24T06:18:52.328180+0000","last_clean":"2023-11-24T06:18:52.328180+0000","last_became_active":"2023-11-24T06:18:50.031929+0000","last_became_peered":"2023-11-24T06:18:50.031929+0000","last_unstale":"2023-11-24T06:18:52.328180+0000","last_undegraded":"2023-11-24T06:18:52.328180+0000","last_fullsized":"2023-11-24T06:18:52.328180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:48.987366+0000","last_clean_scrub_stamp":"2023-11-24T06:18:48.987366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:07.064042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.423}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.554}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0489999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0920000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.198}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:18:59.130 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-24T06:18:59.130 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-24T06:18:59.130 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-24T06:18:59.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-24T06:18:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.282+0000 7fb92b1db700 1 Processor -- start 2023-11-24T06:18:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.282+0000 7fb92b1db700 1 -- start start 2023-11-24T06:18:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb92b1db700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9240b0ba0 0x7fb9240af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb92b1db700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9240b0f80 0x7fb9240cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb92b1db700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 0x7fb9240ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb92b1db700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9240b2bd0 con 0x7fb9240b0f80 2023-11-24T06:18:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb92b1db700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9240b1fc0 con 0x7fb9240b0ba0 2023-11-24T06:18:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb92b1db700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9240b22a0 con 0x7fb92415ee90 2023-11-24T06:18:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9240b0f80 0x7fb9240cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9240b0f80 0x7fb9240cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35490/0 (socket says 172.21.15.89:35490) 2023-11-24T06:18:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 -- 172.21.15.89:0/1668021883 learned_addr learned my addr 172.21.15.89:0/1668021883 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb928f77700 1 --2- 172.21.15.89:0/1668021883 >> v2:172.21.15.123:3300/0 conn(0x7fb9240b0ba0 0x7fb9240af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 -- 172.21.15.89:0/1668021883 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 msgr2=0x7fb9240ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 --2- 172.21.15.89:0/1668021883 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 0x7fb9240ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 -- 172.21.15.89:0/1668021883 >> v2:172.21.15.123:3300/0 conn(0x7fb9240b0ba0 msgr2=0x7fb9240af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 --2- 172.21.15.89:0/1668021883 >> v2:172.21.15.123:3300/0 conn(0x7fb9240b0ba0 0x7fb9240af5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 -- 172.21.15.89:0/1668021883 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9240b2580 con 0x7fb9240b0f80 2023-11-24T06:18:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb923fff700 1 --2- 172.21.15.89:0/1668021883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cc760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb9240d1340 tx=0x7fb910009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fc4c790010c1ad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb9237fe700 1 -- 172.21.15.89:0/1668021883 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91000b870 con 0x7fb9240b0f80 2023-11-24T06:18:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb9237fe700 1 -- 172.21.15.89:0/1668021883 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91000b9d0 con 0x7fb9240b0f80 2023-11-24T06:18:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.283+0000 7fb9237fe700 1 -- 172.21.15.89:0/1668021883 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91000bd60 con 0x7fb9240b0f80 2023-11-24T06:18:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 -- 172.21.15.89:0/1668021883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 msgr2=0x7fb9240cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 --2- 172.21.15.89:0/1668021883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cc760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb9240d1340 tx=0x7fb910009bd0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 -- 172.21.15.89:0/1668021883 shutdown_connections 2023-11-24T06:18:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 --2- 172.21.15.89:0/1668021883 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 0x7fb9240ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 --2- 172.21.15.89:0/1668021883 >> v2:172.21.15.123:3300/0 conn(0x7fb9240b0ba0 0x7fb9240af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 --2- 172.21.15.89:0/1668021883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cc760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 -- 172.21.15.89:0/1668021883 >> 172.21.15.89:0/1668021883 conn(0x7fb9240b57b0 msgr2=0x7fb9240b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 -- 172.21.15.89:0/1668021883 shutdown_connections 2023-11-24T06:18:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.284+0000 7fb92b1db700 1 -- 172.21.15.89:0/1668021883 wait complete. 2023-11-24T06:18:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.285+0000 7fb92b1db700 1 Processor -- start 2023-11-24T06:18:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.285+0000 7fb92b1db700 1 -- start start 2023-11-24T06:18:59.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb92b1db700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9240b0ba0 0x7fb924156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb92b1db700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cf830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb92b1db700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 0x7fb9240cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb92b1db700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241f0b60 con 0x7fb9240b0f80 2023-11-24T06:18:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb92b1db700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb924156d50 con 0x7fb9240b0ba0 2023-11-24T06:18:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb92b1db700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9240d03a0 con 0x7fb92415ee90 2023-11-24T06:18:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb923fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb923fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35498/0 (socket says 172.21.15.89:35498) 2023-11-24T06:18:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb923fff700 1 -- 172.21.15.89:0/2240563681 learned_addr learned my addr 172.21.15.89:0/2240563681 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb928f77700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9240b0ba0 0x7fb924156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb929778700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 0x7fb9240cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb923fff700 1 -- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 msgr2=0x7fb9240cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb923fff700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 0x7fb9240cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb923fff700 1 -- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9240b0ba0 msgr2=0x7fb924156810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb923fff700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9240b0ba0 0x7fb924156810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb923fff700 1 -- 172.21.15.89:0/2240563681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb910009550 con 0x7fb9240b0f80 2023-11-24T06:18:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.286+0000 7fb929778700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 0x7fb9240cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:18:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.287+0000 7fb923fff700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cf830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9240722a0 tx=0x7fb910004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.287+0000 7fb9217fa700 1 -- 172.21.15.89:0/2240563681 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910004310 con 0x7fb9240b0f80 2023-11-24T06:18:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.287+0000 7fb9217fa700 1 -- 172.21.15.89:0/2240563681 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91001f070 con 0x7fb9240b0f80 2023-11-24T06:18:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.287+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9240d0680 con 0x7fb9240b0f80 2023-11-24T06:18:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.287+0000 7fb9217fa700 1 -- 172.21.15.89:0/2240563681 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100036a0 con 0x7fb9240b0f80 2023-11-24T06:18:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.287+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9240ad550 con 0x7fb9240b0f80 2023-11-24T06:18:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.288+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb924067f60 con 0x7fb9240b0f80 2023-11-24T06:18:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.288+0000 7fb9217fa700 1 -- 172.21.15.89:0/2240563681 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb910021030 con 0x7fb9240b0f80 2023-11-24T06:18:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.289+0000 7fb9217fa700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb90806aed0 0x7fb90806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.289+0000 7fb928f77700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb90806aed0 0x7fb90806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.289+0000 7fb9217fa700 1 -- 172.21.15.89:0/2240563681 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb91008aa30 con 0x7fb9240b0f80 2023-11-24T06:18:59.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.289+0000 7fb928f77700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb90806aed0 0x7fb90806d390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb92409e0c0 tx=0x7fb918008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:59.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.293+0000 7fb9217fa700 1 -- 172.21.15.89:0/2240563681 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb910055c90 con 0x7fb9240b0f80 2023-11-24T06:18:59.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.527+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb924071f00 con 0x7fb9240b0f80 2023-11-24T06:18:59.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.527+0000 7fb9217fa700 1 -- 172.21.15.89:0/2240563681 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb9100592b0 con 0x7fb9240b0f80 2023-11-24T06:18:59.529 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-24T06:18:59.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb90806aed0 msgr2=0x7fb90806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb90806aed0 0x7fb90806d390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb92409e0c0 tx=0x7fb918008040 comp rx=0 tx=0).stop 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 msgr2=0x7fb9240cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cf830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9240722a0 tx=0x7fb910004750 comp rx=0 tx=0).stop 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 shutdown_connections 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb92415ee90 0x7fb9240cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb90806aed0 0x7fb90806d390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9240b0ba0 0x7fb924156810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 --2- 172.21.15.89:0/2240563681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9240b0f80 0x7fb9240cf830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 >> 172.21.15.89:0/2240563681 conn(0x7fb9240b57b0 msgr2=0x7fb9241586b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 shutdown_connections 2023-11-24T06:18:59.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.529+0000 7fb92b1db700 1 -- 172.21.15.89:0/2240563681 wait complete. 2023-11-24T06:18:59.543 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-24T06:18:59.544 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:18:59.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.661+0000 7fb3ca8dc700 1 Processor -- start 2023-11-24T06:18:59.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.661+0000 7fb3ca8dc700 1 -- start start 2023-11-24T06:18:59.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3ca8dc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb3c40adf00 0x7fb3c406c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3ca8dc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 0x7fb3c40b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3ca8dc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c40af950 con 0x7fb3c41446a0 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3ca8dc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c4144a80 con 0x7fb3c40adf00 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3ca8dc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c406c9c0 con 0x7fb3c406c540 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c37fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb3c40adf00 0x7fb3c406c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c37fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb3c40adf00 0x7fb3c406c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35506/0 (socket says 172.21.15.89:35506) 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c37fe700 1 -- 172.21.15.89:0/3958783434 learned_addr learned my addr 172.21.15.89:0/3958783434 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c3fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb3c41446a0 0x7fb3c40b5ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39008/0 (socket says 172.21.15.89:39008) 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c8e79700 1 --2- 172.21.15.89:0/3958783434 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 0x7fb3c40b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c37fe700 1 -- 172.21.15.89:0/3958783434 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 msgr2=0x7fb3c40b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:18:59.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c37fe700 1 --2- 172.21.15.89:0/3958783434 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 0x7fb3c40b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c37fe700 1 -- 172.21.15.89:0/3958783434 >> v1:172.21.15.89:6789/0 conn(0x7fb3c41446a0 legacy=0x7fb3c40b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c2ffd700 1 -- 172.21.15.89:0/3958783434 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400198362 0 0) 0x7fb3c40af950 con 0x7fb3c41446a0 2023-11-24T06:18:59.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.662+0000 7fb3c37fe700 1 -- 172.21.15.89:0/3958783434 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c40afc30 con 0x7fb3c40adf00 2023-11-24T06:18:59.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3c37fe700 1 --2- 172.21.15.89:0/3958783434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c406c000 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40098f0 tx=0x7fb3b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1f56c52a778aaf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:59.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3c2ffd700 1 -- 172.21.15.89:0/3958783434 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b400b8a0 con 0x7fb3c40adf00 2023-11-24T06:18:59.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3c2ffd700 1 -- 172.21.15.89:0/3958783434 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b400ba00 con 0x7fb3c40adf00 2023-11-24T06:18:59.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3c2ffd700 1 -- 172.21.15.89:0/3958783434 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b400bd90 con 0x7fb3c40adf00 2023-11-24T06:18:59.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/3958783434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 msgr2=0x7fb3c406c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/3958783434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c406c000 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40098f0 tx=0x7fb3b4009c00 comp rx=0 tx=0).stop 2023-11-24T06:18:59.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/3958783434 shutdown_connections 2023-11-24T06:18:59.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/3958783434 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 0x7fb3c40b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/3958783434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c406c000 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/3958783434 >> 172.21.15.89:0/3958783434 conn(0x7fb3c4071b90 msgr2=0x7fb3c4071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:59.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/3958783434 shutdown_connections 2023-11-24T06:18:59.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.663+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/3958783434 wait complete. 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.664+0000 7fb3ca8dc700 1 Processor -- start 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3ca8dc700 1 -- start start 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3ca8dc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 0x7fb3c40cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3ca8dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c4159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3ca8dc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c41446a0 0x7fb3c4155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3ca8dc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c40ce3b0 con 0x7fb3c40adf00 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3ca8dc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4156770 con 0x7fb3c41446a0 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3ca8dc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4156240 con 0x7fb3c406c540 2023-11-24T06:18:59.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3c37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c4159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3c37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c4159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35512/0 (socket says 172.21.15.89:35512) 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.665+0000 7fb3c37fe700 1 -- 172.21.15.89:0/1795879400 learned_addr learned my addr 172.21.15.89:0/1795879400 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c3fff700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 0x7fb3c40cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c8e79700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c41446a0 0x7fb3c4155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c37fe700 1 -- 172.21.15.89:0/1795879400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 msgr2=0x7fb3c40cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c37fe700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 0x7fb3c40cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c37fe700 1 -- 172.21.15.89:0/1795879400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c41446a0 msgr2=0x7fb3c4155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c37fe700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c41446a0 0x7fb3c4155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c37fe700 1 -- 172.21.15.89:0/1795879400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b4009550 con 0x7fb3c40adf00 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c37fe700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c4159ea0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009c40 tx=0x7fb3b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c0ff9700 1 -- 172.21.15.89:0/1795879400 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4004510 con 0x7fb3c40adf00 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c0ff9700 1 -- 172.21.15.89:0/1795879400 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b40036a0 con 0x7fb3c40adf00 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3c0ff9700 1 -- 172.21.15.89:0/1795879400 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4003a10 con 0x7fb3c40adf00 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c40cd7a0 con 0x7fb3c40adf00 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.666+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c40cdd60 con 0x7fb3c40adf00 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.667+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c40753f0 con 0x7fb3c40adf00 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.667+0000 7fb3c0ff9700 1 -- 172.21.15.89:0/1795879400 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb3b4003b70 con 0x7fb3c40adf00 2023-11-24T06:18:59.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.668+0000 7fb3c0ff9700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb3a406aed0 0x7fb3a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:18:59.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.668+0000 7fb3c3fff700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb3a406aed0 0x7fb3a406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:18:59.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.668+0000 7fb3c0ff9700 1 -- 172.21.15.89:0/1795879400 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3b4021030 con 0x7fb3c40adf00 2023-11-24T06:18:59.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.668+0000 7fb3c3fff700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb3a406aed0 0x7fb3a406d390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac000f60 tx=0x7fb3ac007e70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:18:59.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.672+0000 7fb3c0ff9700 1 -- 172.21.15.89:0/1795879400 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3b4055d00 con 0x7fb3c40adf00 2023-11-24T06:18:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.861+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3c4141a90 con 0x7fb3c40adf00 2023-11-24T06:18:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.862+0000 7fb3c0ff9700 1 -- 172.21.15.89:0/1795879400 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7fb3b4015090 con 0x7fb3c40adf00 2023-11-24T06:18:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:18:59.864 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:18:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb3a406aed0 msgr2=0x7fb3a406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb3a406aed0 0x7fb3a406d390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac000f60 tx=0x7fb3ac007e70 comp rx=0 tx=0).stop 2023-11-24T06:18:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 msgr2=0x7fb3c4159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:18:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c4159ea0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009c40 tx=0x7fb3b4004750 comp rx=0 tx=0).stop 2023-11-24T06:18:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 shutdown_connections 2023-11-24T06:18:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb3a406aed0 0x7fb3a406d390 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c41446a0 0x7fb3c4155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb3c406c540 0x7fb3c40cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 --2- 172.21.15.89:0/1795879400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c40adf00 0x7fb3c4159ea0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:18:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 >> 172.21.15.89:0/1795879400 conn(0x7fb3c4071b90 msgr2=0x7fb3c406df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:18:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 shutdown_connections 2023-11-24T06:18:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.864+0000 7fb3ca8dc700 1 -- 172.21.15.89:0/1795879400 wait complete. 2023-11-24T06:18:59.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:18:59.878 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:00.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.999+0000 7fa96d01b700 1 Processor -- start 2023-11-24T06:19:00.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:18:59.999+0000 7fa96d01b700 1 -- start start 2023-11-24T06:19:00.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96d01b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa96815b370 0x7fa96815b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96d01b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9680b3c60 0x7fa9680b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96d01b700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9680ae5e0 con 0x7fa9681587f0 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96d01b700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9680b4610 con 0x7fa9680b3c60 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96d01b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96815be30 con 0x7fa96815b370 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96759e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9680b3c60 0x7fa9680b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96759e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9680b3c60 0x7fa9680b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35522/0 (socket says 172.21.15.89:35522) 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96759e700 1 -- 172.21.15.89:0/401170315 learned_addr learned my addr 172.21.15.89:0/401170315 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96659c700 1 --2- 172.21.15.89:0/401170315 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa96815b370 0x7fa96815b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96759e700 1 -- 172.21.15.89:0/401170315 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa96815b370 msgr2=0x7fa96815b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96759e700 1 --2- 172.21.15.89:0/401170315 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa96815b370 0x7fa96815b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96759e700 1 -- 172.21.15.89:0/401170315 >> v1:172.21.15.123:6789/0 conn(0x7fa9681587f0 legacy=0x7fa96806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.000+0000 7fa96759e700 1 -- 172.21.15.89:0/401170315 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96815c110 con 0x7fa9680b3c60 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96759e700 1 --2- 172.21.15.89:0/401170315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9680b3c60 0x7fa9680b40d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa96000d150 tx=0x7fa96000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6feb22e3ad11c6f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa965d9b700 1 -- 172.21.15.89:0/401170315 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960010070 con 0x7fa9680b3c60 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa965d9b700 1 -- 172.21.15.89:0/401170315 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96000dd80 con 0x7fa9680b3c60 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa965d9b700 1 -- 172.21.15.89:0/401170315 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96000b310 con 0x7fa9680b3c60 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96d01b700 1 -- 172.21.15.89:0/401170315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9680b3c60 msgr2=0x7fa9680b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96d01b700 1 --2- 172.21.15.89:0/401170315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9680b3c60 0x7fa9680b40d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa96000d150 tx=0x7fa96000d460 comp rx=0 tx=0).stop 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96d01b700 1 -- 172.21.15.89:0/401170315 shutdown_connections 2023-11-24T06:19:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96d01b700 1 --2- 172.21.15.89:0/401170315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9680b3c60 0x7fa9680b40d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96d01b700 1 --2- 172.21.15.89:0/401170315 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa96815b370 0x7fa96815b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96d01b700 1 -- 172.21.15.89:0/401170315 >> 172.21.15.89:0/401170315 conn(0x7fa9680d0bf0 msgr2=0x7fa9680d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96d01b700 1 -- 172.21.15.89:0/401170315 shutdown_connections 2023-11-24T06:19:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.001+0000 7fa96d01b700 1 -- 172.21.15.89:0/401170315 wait complete. 2023-11-24T06:19:00.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.003+0000 7fa96d01b700 1 Processor -- start 2023-11-24T06:19:00.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.003+0000 7fa96d01b700 1 -- start start 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.003+0000 7fa96d01b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9680b3c60 0x7fa96809e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.003+0000 7fa96d01b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9681587f0 0x7fa968146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.003+0000 7fa96d01b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96815b370 0x7fa96809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.003+0000 7fa96d01b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96809e350 con 0x7fa96815b370 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.003+0000 7fa96d01b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96809d470 con 0x7fa9681587f0 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.003+0000 7fa96d01b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96809d710 con 0x7fa9680b3c60 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96815b370 0x7fa96809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96815b370 0x7fa96809aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35528/0 (socket says 172.21.15.89:35528) 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 -- 172.21.15.89:0/1212479469 learned_addr learned my addr 172.21.15.89:0/1212479469 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa966d9d700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9680b3c60 0x7fa96809e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96659c700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9681587f0 0x7fa968146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 -- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9680b3c60 msgr2=0x7fa96809e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9680b3c60 0x7fa96809e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 -- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9681587f0 msgr2=0x7fa968146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9681587f0 0x7fa968146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 -- 172.21.15.89:0/1212479469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96000bd40 con 0x7fa96815b370 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96659c700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9681587f0 0x7fa968146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa966d9d700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9680b3c60 0x7fa96809e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96759e700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96815b370 0x7fa96809aa00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa9680d1730 tx=0x7fa960004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa94f7fe700 1 -- 172.21.15.89:0/1212479469 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96000ab20 con 0x7fa96815b370 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa94f7fe700 1 -- 172.21.15.89:0/1212479469 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96000ac80 con 0x7fa96815b370 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa94f7fe700 1 -- 172.21.15.89:0/1212479469 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960004030 con 0x7fa96815b370 2023-11-24T06:19:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96809d9b0 con 0x7fa96815b370 2023-11-24T06:19:00.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.004+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96809df70 con 0x7fa96815b370 2023-11-24T06:19:00.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.005+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96809e490 con 0x7fa96815b370 2023-11-24T06:19:00.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.006+0000 7fa94f7fe700 1 -- 172.21.15.89:0/1212479469 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa960004190 con 0x7fa96815b370 2023-11-24T06:19:00.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.006+0000 7fa94f7fe700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa95406aed0 0x7fa95406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.006+0000 7fa94f7fe700 1 -- 172.21.15.89:0/1212479469 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa96008cdd0 con 0x7fa96815b370 2023-11-24T06:19:00.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.006+0000 7fa966d9d700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa95406aed0 0x7fa95406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.006+0000 7fa966d9d700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa95406aed0 0x7fa95406d390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa958003fc0 tx=0x7fa958009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.010+0000 7fa94f7fe700 1 -- 172.21.15.89:0/1212479469 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa96809e490 con 0x7fa96815b370 2023-11-24T06:19:00.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.200+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa96809e490 con 0x7fa96815b370 2023-11-24T06:19:00.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.201+0000 7fa94f7fe700 1 -- 172.21.15.89:0/1212479469 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa960019020 con 0x7fa96815b370 2023-11-24T06:19:00.203 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:00.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.202+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa95406aed0 msgr2=0x7fa95406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa95406aed0 0x7fa95406d390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa958003fc0 tx=0x7fa958009040 comp rx=0 tx=0).stop 2023-11-24T06:19:00.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96815b370 msgr2=0x7fa96809aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96815b370 0x7fa96809aa00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa9680d1730 tx=0x7fa960004ad0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 shutdown_connections 2023-11-24T06:19:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96815b370 0x7fa96809aa00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9680b3c60 0x7fa96809e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa95406aed0 0x7fa95406d390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 --2- 172.21.15.89:0/1212479469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9681587f0 0x7fa968146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 >> 172.21.15.89:0/1212479469 conn(0x7fa9680d0bf0 msgr2=0x7fa968156910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 shutdown_connections 2023-11-24T06:19:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.203+0000 7fa96d01b700 1 -- 172.21.15.89:0/1212479469 wait complete. 2023-11-24T06:19:00.217 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.337+0000 7ff69300d700 1 Processor -- start 2023-11-24T06:19:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.337+0000 7ff69300d700 1 -- start start 2023-11-24T06:19:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.338+0000 7ff69300d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff68c0c6c30 0x7ff68c0cea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.338+0000 7ff69300d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff68c0cef40 0x7ff68c0cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.338+0000 7ff69300d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68c0d0140 con 0x7ff68c15bf50 2023-11-24T06:19:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.338+0000 7ff69300d700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c15c330 con 0x7ff68c0cef40 2023-11-24T06:19:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.338+0000 7ff69300d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c0cf3d0 con 0x7ff68c0c6c30 2023-11-24T06:19:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff6915aa700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff68c0cef40 0x7ff68c0cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff690da9700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff68c15bf50 0x7ff68c0ce2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39036/0 (socket says 172.21.15.89:39036) 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff690da9700 1 -- 172.21.15.89:0/3870366876 learned_addr learned my addr 172.21.15.89:0/3870366876 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff68bfff700 1 --2- 172.21.15.89:0/3870366876 >> v2:172.21.15.123:3300/0 conn(0x7ff68c0c6c30 0x7ff68c0cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff6915aa700 1 -- 172.21.15.89:0/3870366876 >> v2:172.21.15.123:3300/0 conn(0x7ff68c0c6c30 msgr2=0x7ff68c0cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff6915aa700 1 --2- 172.21.15.89:0/3870366876 >> v2:172.21.15.123:3300/0 conn(0x7ff68c0c6c30 0x7ff68c0cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff6915aa700 1 -- 172.21.15.89:0/3870366876 >> v1:172.21.15.89:6789/0 conn(0x7ff68c15bf50 legacy=0x7ff68c0ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff6915aa700 1 -- 172.21.15.89:0/3870366876 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c0d03e0 con 0x7ff68c0cef40 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff68b7fe700 1 -- 172.21.15.89:0/3870366876 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541017676 0 0) 0x7ff68c0d0140 con 0x7ff68c15bf50 2023-11-24T06:19:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff6915aa700 1 --2- 172.21.15.89:0/3870366876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c0cef40 0x7ff68c0cc6c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff68c072420 tx=0x7ff68400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9563c0a13d02e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff68b7fe700 1 -- 172.21.15.89:0/3870366876 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68400e070 con 0x7ff68c0cef40 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff68b7fe700 1 -- 172.21.15.89:0/3870366876 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68400bbd0 con 0x7ff68c0cef40 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.339+0000 7ff68b7fe700 1 -- 172.21.15.89:0/3870366876 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684012430 con 0x7ff68c0cef40 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.340+0000 7ff69300d700 1 -- 172.21.15.89:0/3870366876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c0cef40 msgr2=0x7ff68c0cc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.340+0000 7ff69300d700 1 --2- 172.21.15.89:0/3870366876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c0cef40 0x7ff68c0cc6c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff68c072420 tx=0x7ff68400b330 comp rx=0 tx=0).stop 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.340+0000 7ff69300d700 1 -- 172.21.15.89:0/3870366876 shutdown_connections 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.340+0000 7ff69300d700 1 --2- 172.21.15.89:0/3870366876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c0cef40 0x7ff68c0cc6c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.340+0000 7ff69300d700 1 --2- 172.21.15.89:0/3870366876 >> v2:172.21.15.123:3300/0 conn(0x7ff68c0c6c30 0x7ff68c0cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.340+0000 7ff69300d700 1 -- 172.21.15.89:0/3870366876 >> 172.21.15.89:0/3870366876 conn(0x7ff68c0b3780 msgr2=0x7ff68c0b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.340+0000 7ff69300d700 1 -- 172.21.15.89:0/3870366876 shutdown_connections 2023-11-24T06:19:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.340+0000 7ff69300d700 1 -- 172.21.15.89:0/3870366876 wait complete. 2023-11-24T06:19:00.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.341+0000 7ff69300d700 1 Processor -- start 2023-11-24T06:19:00.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.341+0000 7ff69300d700 1 -- start start 2023-11-24T06:19:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff69300d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68c0c6c30 0x7ff68c1f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff69300d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff68c0cef40 0x7ff68c06fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff69300d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c15bf50 0x7ff68c155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff69300d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c155860 con 0x7ff68c15bf50 2023-11-24T06:19:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff69300d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c158f40 con 0x7ff68c0c6c30 2023-11-24T06:19:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff69300d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c158980 con 0x7ff68c0cef40 2023-11-24T06:19:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c15bf50 0x7ff68c155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c15bf50 0x7ff68c155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35542/0 (socket says 172.21.15.89:35542) 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 -- 172.21.15.89:0/2188037554 learned_addr learned my addr 172.21.15.89:0/2188037554 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff690da9700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68c0c6c30 0x7ff68c1f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff68bfff700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff68c0cef40 0x7ff68c06fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 -- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff68c0cef40 msgr2=0x7ff68c06fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff68c0cef40 0x7ff68c06fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 -- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68c0c6c30 msgr2=0x7ff68c1f18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68c0c6c30 0x7ff68c1f18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 -- 172.21.15.89:0/2188037554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff684009a70 con 0x7ff68c15bf50 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff690da9700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68c0c6c30 0x7ff68c1f18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6915aa700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c15bf50 0x7ff68c155b60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff68400b730 tx=0x7ff684004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6897fa700 1 -- 172.21.15.89:0/2188037554 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684012430 con 0x7ff68c15bf50 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.342+0000 7ff6897fa700 1 -- 172.21.15.89:0/2188037554 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff684020070 con 0x7ff68c15bf50 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.343+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c158c60 con 0x7ff68c15bf50 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.343+0000 7ff6897fa700 1 -- 172.21.15.89:0/2188037554 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684004280 con 0x7ff68c15bf50 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.343+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68c0ca4e0 con 0x7ff68c15bf50 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.343+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68c0ca6a0 con 0x7ff68c15bf50 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.344+0000 7ff6897fa700 1 -- 172.21.15.89:0/2188037554 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff6840043e0 con 0x7ff68c15bf50 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.344+0000 7ff6897fa700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff67006aed0 0x7ff67006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.344+0000 7ff6897fa700 1 -- 172.21.15.89:0/2188037554 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff684022070 con 0x7ff68c15bf50 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.344+0000 7ff690da9700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff67006aed0 0x7ff67006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.345+0000 7ff690da9700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff67006aed0 0x7ff67006d390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff680000f60 tx=0x7ff680007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.349+0000 7ff6897fa700 1 -- 172.21.15.89:0/2188037554 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff684056680 con 0x7ff68c15bf50 2023-11-24T06:19:00.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.538+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff68c0ca060 con 0x7ff68c15bf50 2023-11-24T06:19:00.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.539+0000 7ff6897fa700 1 -- 172.21.15.89:0/2188037554 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff68c0ca060 con 0x7ff68c15bf50 2023-11-24T06:19:00.541 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:00.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff67006aed0 msgr2=0x7ff67006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff67006aed0 0x7ff67006d390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff680000f60 tx=0x7ff680007d50 comp rx=0 tx=0).stop 2023-11-24T06:19:00.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c15bf50 msgr2=0x7ff68c155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c15bf50 0x7ff68c155b60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff68400b730 tx=0x7ff684004ad0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 shutdown_connections 2023-11-24T06:19:00.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff68c15bf50 0x7ff68c155b60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff67006aed0 0x7ff67006d390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68c0c6c30 0x7ff68c1f18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 --2- 172.21.15.89:0/2188037554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff68c0cef40 0x7ff68c06fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 >> 172.21.15.89:0/2188037554 conn(0x7ff68c0b3780 msgr2=0x7ff68c06ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:00.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 shutdown_connections 2023-11-24T06:19:00.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.541+0000 7ff69300d700 1 -- 172.21.15.89:0/2188037554 wait complete. 2023-11-24T06:19:00.555 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:00.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.675+0000 7fcb25321700 1 Processor -- start 2023-11-24T06:19:00.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.675+0000 7fcb25321700 1 -- start start 2023-11-24T06:19:00.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb25321700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb2015e140 0x7fcb200d3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb25321700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcb200d39d0 0x7fcb200d4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb25321700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200b80e0 con 0x7fcb200b98c0 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb25321700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2015e520 con 0x7fcb200d39d0 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb25321700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200d2930 con 0x7fcb2015e140 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb1f7fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcb200b98c0 0x7fcb200b59b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39052/0 (socket says 172.21.15.89:39052) 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb1f7fe700 1 -- 172.21.15.89:0/1966127724 learned_addr learned my addr 172.21.15.89:0/1966127724 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb1e7fc700 1 --2- 172.21.15.89:0/1966127724 >> v2:172.21.15.123:3300/0 conn(0x7fcb200d39d0 0x7fcb200d4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb1effd700 1 --2- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb2015e140 0x7fcb200d3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb1dffb700 1 -- 172.21.15.89:0/1966127724 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384593312 0 0) 0x7fcb18001490 con 0x7fcb200b98c0 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.676+0000 7fcb1dffb700 1 -- 172.21.15.89:0/1966127724 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb080035b0 con 0x7fcb200b98c0 2023-11-24T06:19:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1e7fc700 1 -- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb2015e140 msgr2=0x7fcb200d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1e7fc700 1 --2- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb2015e140 0x7fcb200d3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1e7fc700 1 -- 172.21.15.89:0/1966127724 >> v1:172.21.15.89:6789/0 conn(0x7fcb200b98c0 legacy=0x7fcb200b59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1e7fc700 1 -- 172.21.15.89:0/1966127724 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200d2c10 con 0x7fcb200d39d0 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1effd700 1 --2- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb2015e140 0x7fcb200d3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1e7fc700 1 --2- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb200d39d0 0x7fcb200d4400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009840 tx=0x7fcb10009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55688e89b12b3019 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1dffb700 1 -- 172.21.15.89:0/1966127724 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10011830 con 0x7fcb200d39d0 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1dffb700 1 -- 172.21.15.89:0/1966127724 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10011990 con 0x7fcb200d39d0 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb1dffb700 1 -- 172.21.15.89:0/1966127724 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10011d20 con 0x7fcb200d39d0 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb25321700 1 -- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb200d39d0 msgr2=0x7fcb200d4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.677+0000 7fcb25321700 1 --2- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb200d39d0 0x7fcb200d4400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009840 tx=0x7fcb10009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.678+0000 7fcb25321700 1 -- 172.21.15.89:0/1966127724 shutdown_connections 2023-11-24T06:19:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.678+0000 7fcb25321700 1 --2- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb2015e140 0x7fcb200d3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.678+0000 7fcb25321700 1 --2- 172.21.15.89:0/1966127724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb200d39d0 0x7fcb200d4400 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.678+0000 7fcb25321700 1 -- 172.21.15.89:0/1966127724 >> 172.21.15.89:0/1966127724 conn(0x7fcb2006b410 msgr2=0x7fcb2006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:00.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.678+0000 7fcb25321700 1 -- 172.21.15.89:0/1966127724 shutdown_connections 2023-11-24T06:19:00.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.678+0000 7fcb25321700 1 -- 172.21.15.89:0/1966127724 wait complete. 2023-11-24T06:19:00.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.679+0000 7fcb25321700 1 Processor -- start 2023-11-24T06:19:00.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.679+0000 7fcb25321700 1 -- start start 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb25321700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb200b98c0 0x7fcb201f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb25321700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb200d39d0 0x7fcb200c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb25321700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb2015e140 0x7fcb200c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb25321700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200c8d80 con 0x7fcb200d39d0 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb25321700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200ca600 con 0x7fcb2015e140 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb25321700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200b4af0 con 0x7fcb200b98c0 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb200d39d0 0x7fcb200c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb200d39d0 0x7fcb200c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35554/0 (socket says 172.21.15.89:35554) 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 -- 172.21.15.89:0/76294804 learned_addr learned my addr 172.21.15.89:0/76294804 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1effd700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb200b98c0 0x7fcb201f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1f7fe700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb2015e140 0x7fcb200c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 -- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb200b98c0 msgr2=0x7fcb201f30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb200b98c0 0x7fcb201f30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 -- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb2015e140 msgr2=0x7fcb200c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb2015e140 0x7fcb200c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 -- 172.21.15.89:0/76294804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb140094a0 con 0x7fcb200d39d0 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1effd700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb200b98c0 0x7fcb201f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb1e7fc700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb200d39d0 0x7fcb200c8210 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcb100090b0 tx=0x7fcb10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb077fe700 1 -- 172.21.15.89:0/76294804 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb100036a0 con 0x7fcb200d39d0 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb077fe700 1 -- 172.21.15.89:0/76294804 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1001f070 con 0x7fcb200d39d0 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb077fe700 1 -- 172.21.15.89:0/76294804 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10003a00 con 0x7fcb200d39d0 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb100094a0 con 0x7fcb200d39d0 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.680+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb200b50b0 con 0x7fcb200d39d0 2023-11-24T06:19:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.681+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb20068fb0 con 0x7fcb200d39d0 2023-11-24T06:19:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.682+0000 7fcb077fe700 1 -- 172.21.15.89:0/76294804 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcb10003b60 con 0x7fcb200d39d0 2023-11-24T06:19:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.682+0000 7fcb077fe700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcb0c06aed0 0x7fcb0c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.682+0000 7fcb077fe700 1 -- 172.21.15.89:0/76294804 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb10021070 con 0x7fcb200d39d0 2023-11-24T06:19:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.682+0000 7fcb1effd700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcb0c06aed0 0x7fcb0c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.683+0000 7fcb1effd700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcb0c06aed0 0x7fcb0c06d390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009c50 tx=0x7fcb14008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:00.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.687+0000 7fcb077fe700 1 -- 172.21.15.89:0/76294804 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb10055fb0 con 0x7fcb200d39d0 2023-11-24T06:19:00.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.879+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb2015fa50 con 0x7fcb200d39d0 2023-11-24T06:19:00.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.880+0000 7fcb077fe700 1 -- 172.21.15.89:0/76294804 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb1002b020 con 0x7fcb200d39d0 2023-11-24T06:19:00.884 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:00.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.881+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcb0c06aed0 msgr2=0x7fcb0c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.881+0000 7fcb25321700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcb0c06aed0 0x7fcb0c06d390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009c50 tx=0x7fcb14008040 comp rx=0 tx=0).stop 2023-11-24T06:19:00.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.881+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb200d39d0 msgr2=0x7fcb200c8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:00.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.881+0000 7fcb25321700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb200d39d0 0x7fcb200c8210 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcb100090b0 tx=0x7fcb10004750 comp rx=0 tx=0).stop 2023-11-24T06:19:00.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.882+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 shutdown_connections 2023-11-24T06:19:00.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.882+0000 7fcb25321700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcb0c06aed0 0x7fcb0c06d390 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.882+0000 7fcb25321700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb2015e140 0x7fcb200c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.882+0000 7fcb25321700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb200b98c0 0x7fcb201f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.882+0000 7fcb25321700 1 --2- 172.21.15.89:0/76294804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb200d39d0 0x7fcb200c8210 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:00.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.882+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 >> 172.21.15.89:0/76294804 conn(0x7fcb2006b410 msgr2=0x7fcb2015f340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:00.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.882+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 shutdown_connections 2023-11-24T06:19:00.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:00.882+0000 7fcb25321700 1 -- 172.21.15.89:0/76294804 wait complete. 2023-11-24T06:19:00.896 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.013+0000 7fe6eac4b700 1 Processor -- start 2023-11-24T06:19:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.013+0000 7fe6eac4b700 1 -- start start 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6eac4b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e415ae20 0x7fe6e40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6eac4b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6e40b1a70 0x7fe6e4164b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6eac4b700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e40cfd40 con 0x7fe6e415b200 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6eac4b700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e40d0020 con 0x7fe6e40b1a70 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6eac4b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e40affa0 con 0x7fe6e415ae20 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6e91e8700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6e40b1a70 0x7fe6e4164b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6e3fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe6e415b200 0x7fe6e40b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39066/0 (socket says 172.21.15.89:39066) 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6e3fff700 1 -- 172.21.15.89:0/3373836395 learned_addr learned my addr 172.21.15.89:0/3373836395 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6e91e8700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6e40b1a70 0x7fe6e4164b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35564/0 (socket says 172.21.15.89:35564) 2023-11-24T06:19:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.014+0000 7fe6e89e7700 1 --2- 172.21.15.89:0/3373836395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e415ae20 0x7fe6e40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e91e8700 1 -- 172.21.15.89:0/3373836395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e415ae20 msgr2=0x7fe6e40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e91e8700 1 --2- 172.21.15.89:0/3373836395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e415ae20 0x7fe6e40b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e91e8700 1 -- 172.21.15.89:0/3373836395 >> v1:172.21.15.89:6789/0 conn(0x7fe6e415b200 legacy=0x7fe6e40b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e91e8700 1 -- 172.21.15.89:0/3373836395 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e40b0280 con 0x7fe6e40b1a70 2023-11-24T06:19:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e37fe700 1 -- 172.21.15.89:0/3373836395 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972102671 0 0) 0x7fe6e40cfd40 con 0x7fe6e415b200 2023-11-24T06:19:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e89e7700 1 --2- 172.21.15.89:0/3373836395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e415ae20 0x7fe6e40b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e91e8700 1 --2- 172.21.15.89:0/3373836395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e40b1a70 0x7fe6e4164b50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc009d90 tx=0x7fe6dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b46e65dca88dec server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e37fe700 1 -- 172.21.15.89:0/3373836395 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc00e070 con 0x7fe6e40b1a70 2023-11-24T06:19:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e37fe700 1 -- 172.21.15.89:0/3373836395 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc00bbd0 con 0x7fe6e40b1a70 2023-11-24T06:19:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6e37fe700 1 -- 172.21.15.89:0/3373836395 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc012430 con 0x7fe6e40b1a70 2023-11-24T06:19:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3373836395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e40b1a70 msgr2=0x7fe6e4164b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3373836395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e40b1a70 0x7fe6e4164b50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc009d90 tx=0x7fe6dc00b330 comp rx=0 tx=0).stop 2023-11-24T06:19:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3373836395 shutdown_connections 2023-11-24T06:19:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3373836395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e40b1a70 0x7fe6e4164b50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3373836395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e415ae20 0x7fe6e40b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3373836395 >> 172.21.15.89:0/3373836395 conn(0x7fe6e40b57b0 msgr2=0x7fe6e40b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.015+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3373836395 shutdown_connections 2023-11-24T06:19:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.016+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3373836395 wait complete. 2023-11-24T06:19:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.017+0000 7fe6eac4b700 1 Processor -- start 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.017+0000 7fe6eac4b700 1 -- start start 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6eac4b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e40b1a70 0x7fe6e40ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6eac4b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e415ae20 0x7fe6e40c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6eac4b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e415b200 0x7fe6e40c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6eac4b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e41f3c10 con 0x7fe6e415ae20 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6eac4b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e41f3ef0 con 0x7fe6e415b200 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6eac4b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e41f41d0 con 0x7fe6e40b1a70 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e415ae20 0x7fe6e40c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e415ae20 0x7fe6e40c9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35568/0 (socket says 172.21.15.89:35568) 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 -- 172.21.15.89:0/3744508097 learned_addr learned my addr 172.21.15.89:0/3744508097 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e89e7700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e40b1a70 0x7fe6e40ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e91e8700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e415b200 0x7fe6e40c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 -- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e40b1a70 msgr2=0x7fe6e40ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e40b1a70 0x7fe6e40ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 -- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e415b200 msgr2=0x7fe6e40c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e415b200 0x7fe6e40c9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 -- 172.21.15.89:0/3744508097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc009a70 con 0x7fe6e415ae20 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e91e8700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e415b200 0x7fe6e40c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e3fff700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e415ae20 0x7fe6e40c9610 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000c550 tx=0x7fe6d000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e17fa700 1 -- 172.21.15.89:0/3744508097 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0004c80 con 0x7fe6e415ae20 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e17fa700 1 -- 172.21.15.89:0/3744508097 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0004e00 con 0x7fe6e415ae20 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e41f44b0 con 0x7fe6e415ae20 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6e17fa700 1 -- 172.21.15.89:0/3744508097 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0007310 con 0x7fe6e415ae20 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.018+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e40cd0f0 con 0x7fe6e415ae20 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.019+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e4067f60 con 0x7fe6e415ae20 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.020+0000 7fe6e17fa700 1 -- 172.21.15.89:0/3744508097 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe6d0007470 con 0x7fe6e415ae20 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.020+0000 7fe6e17fa700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c806aed0 0x7fe6c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.020+0000 7fe6e89e7700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c806aed0 0x7fe6c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.020+0000 7fe6e17fa700 1 -- 172.21.15.89:0/3744508097 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6d0089270 con 0x7fe6e415ae20 2023-11-24T06:19:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.020+0000 7fe6e89e7700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c806aed0 0x7fe6c806d390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80099d0 tx=0x7fe6d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.024+0000 7fe6e17fa700 1 -- 172.21.15.89:0/3744508097 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6d0054320 con 0x7fe6e415ae20 2023-11-24T06:19:01.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.257+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6e40cde80 con 0x7fe6e415ae20 2023-11-24T06:19:01.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.258+0000 7fe6e17fa700 1 -- 172.21.15.89:0/3744508097 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fe6d0057940 con 0x7fe6e415ae20 2023-11-24T06:19:01.261 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:01.262 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:18:49.391199+0000","modified":"2023-11-24T06:18:52.422301+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:19:01.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c806aed0 msgr2=0x7fe6c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c806aed0 0x7fe6c806d390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80099d0 tx=0x7fe6d8008040 comp rx=0 tx=0).stop 2023-11-24T06:19:01.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e415ae20 msgr2=0x7fe6e40c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e415ae20 0x7fe6e40c9610 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000c550 tx=0x7fe6d000c860 comp rx=0 tx=0).stop 2023-11-24T06:19:01.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 shutdown_connections 2023-11-24T06:19:01.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c806aed0 0x7fe6c806d390 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e415b200 0x7fe6e40c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e40b1a70 0x7fe6e40ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 --2- 172.21.15.89:0/3744508097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e415ae20 0x7fe6e40c9610 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 >> 172.21.15.89:0/3744508097 conn(0x7fe6e40b57b0 msgr2=0x7fe6e406e160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:01.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 shutdown_connections 2023-11-24T06:19:01.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.260+0000 7fe6eac4b700 1 -- 172.21.15.89:0/3744508097 wait complete. 2023-11-24T06:19:01.265 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-11-24T06:19:01.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:01.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.393+0000 7f35852df700 1 Processor -- start 2023-11-24T06:19:01.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.393+0000 7f35852df700 1 -- start start 2023-11-24T06:19:01.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.393+0000 7f35852df700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f358015bd30 0x7f35800cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.393+0000 7f35852df700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3580162920 0x7f35800cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.393+0000 7f35852df700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3580161bf0 con 0x7f358015e6e0 2023-11-24T06:19:01.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.393+0000 7f35852df700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35800cff40 con 0x7f3580162920 2023-11-24T06:19:01.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.393+0000 7f35852df700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358015f450 con 0x7f358015bd30 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357e7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3580162920 0x7f35800cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357e7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3580162920 0x7f35800cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35574/0 (socket says 172.21.15.89:35574) 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357e7fc700 1 -- 172.21.15.89:0/2063127759 learned_addr learned my addr 172.21.15.89:0/2063127759 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357f7fe700 1 --1- 172.21.15.89:0/2063127759 >> v1:172.21.15.89:6789/0 conn(0x7f358015e6e0 0x7f358015eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39082/0 (socket says 172.21.15.89:39082) 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357effd700 1 --2- 172.21.15.89:0/2063127759 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f358015bd30 0x7f35800cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357e7fc700 1 -- 172.21.15.89:0/2063127759 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f358015bd30 msgr2=0x7f35800cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357e7fc700 1 --2- 172.21.15.89:0/2063127759 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f358015bd30 0x7f35800cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357e7fc700 1 -- 172.21.15.89:0/2063127759 >> v1:172.21.15.89:6789/0 conn(0x7f358015e6e0 legacy=0x7f358015eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357dffb700 1 -- 172.21.15.89:0/2063127759 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622244853 0 0) 0x7f3580161bf0 con 0x7f358015e6e0 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357e7fc700 1 -- 172.21.15.89:0/2063127759 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3580161e40 con 0x7f3580162920 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357effd700 1 --2- 172.21.15.89:0/2063127759 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f358015bd30 0x7f35800cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357e7fc700 1 --2- 172.21.15.89:0/2063127759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3580162920 0x7f35800cfa00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f35740098f0 tx=0x7f3574009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9a7b95a5e703e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357dffb700 1 -- 172.21.15.89:0/2063127759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357400b8a0 con 0x7f3580162920 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357dffb700 1 -- 172.21.15.89:0/2063127759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357400ba00 con 0x7f3580162920 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.394+0000 7f357dffb700 1 -- 172.21.15.89:0/2063127759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357400bd90 con 0x7f3580162920 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.395+0000 7f35852df700 1 -- 172.21.15.89:0/2063127759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3580162920 msgr2=0x7f35800cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.395+0000 7f35852df700 1 --2- 172.21.15.89:0/2063127759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3580162920 0x7f35800cfa00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f35740098f0 tx=0x7f3574009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.395+0000 7f35852df700 1 -- 172.21.15.89:0/2063127759 shutdown_connections 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.395+0000 7f35852df700 1 --2- 172.21.15.89:0/2063127759 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f358015bd30 0x7f35800cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.395+0000 7f35852df700 1 --2- 172.21.15.89:0/2063127759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3580162920 0x7f35800cfa00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.395+0000 7f35852df700 1 -- 172.21.15.89:0/2063127759 >> 172.21.15.89:0/2063127759 conn(0x7f3580072710 msgr2=0x7f3580072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.395+0000 7f35852df700 1 -- 172.21.15.89:0/2063127759 shutdown_connections 2023-11-24T06:19:01.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.395+0000 7f35852df700 1 -- 172.21.15.89:0/2063127759 wait complete. 2023-11-24T06:19:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.396+0000 7f35852df700 1 Processor -- start 2023-11-24T06:19:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.396+0000 7f35852df700 1 -- start start 2023-11-24T06:19:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f35852df700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f358015bd30 0x7f35800b1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f35852df700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358015e6e0 0x7f35800aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f35852df700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3580162920 0x7f35800af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f35852df700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358015b2a0 con 0x7f358015bd30 2023-11-24T06:19:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f35852df700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35800ad170 con 0x7f358015e6e0 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f35852df700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35800af6a0 con 0x7f3580162920 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f358015bd30 0x7f35800b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f358015bd30 0x7f35800b1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35580/0 (socket says 172.21.15.89:35580) 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 -- 172.21.15.89:0/3773243273 learned_addr learned my addr 172.21.15.89:0/3773243273 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357e7fc700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358015e6e0 0x7f35800aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 -- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3580162920 msgr2=0x7f35800af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357f7fe700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3580162920 0x7f35800af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3580162920 0x7f35800af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 -- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358015e6e0 msgr2=0x7f35800aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358015e6e0 0x7f35800aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 -- 172.21.15.89:0/3773243273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3574009550 con 0x7f358015bd30 2023-11-24T06:19:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357e7fc700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358015e6e0 0x7f35800aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.397+0000 7f357effd700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f358015bd30 0x7f35800b1750 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f357000c750 tx=0x7f357000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.398+0000 7f35677fe700 1 -- 172.21.15.89:0/3773243273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3570011800 con 0x7f358015bd30 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.398+0000 7f35677fe700 1 -- 172.21.15.89:0/3773243273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3570011960 con 0x7f358015bd30 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.398+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35800ad3c0 con 0x7f358015bd30 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.398+0000 7f35677fe700 1 -- 172.21.15.89:0/3773243273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3570011cd0 con 0x7f358015bd30 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.398+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35800ad980 con 0x7f358015bd30 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.399+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35800ccb20 con 0x7f358015bd30 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.399+0000 7f35677fe700 1 -- 172.21.15.89:0/3773243273 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3570021350 con 0x7f358015bd30 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.399+0000 7f35677fe700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f356c06af00 0x7f356c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.399+0000 7f35677fe700 1 -- 172.21.15.89:0/3773243273 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3570089590 con 0x7f358015bd30 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.399+0000 7f357e7fc700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f356c06af00 0x7f356c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.400+0000 7f357e7fc700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f356c06af00 0x7f356c06d3c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f35800ae380 tx=0x7f3574008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.404+0000 7f35677fe700 1 -- 172.21.15.89:0/3773243273 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3570054640 con 0x7f358015bd30 2023-11-24T06:19:01.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.593+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35800751e0 con 0x7f358015bd30 2023-11-24T06:19:01.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.594+0000 7f35677fe700 1 -- 172.21.15.89:0/3773243273 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f3570057c60 con 0x7f358015bd30 2023-11-24T06:19:01.598 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:01.598 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f356c06af00 msgr2=0x7f356c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f356c06af00 0x7f356c06d3c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f35800ae380 tx=0x7f3574008030 comp rx=0 tx=0).stop 2023-11-24T06:19:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f358015bd30 msgr2=0x7f35800b1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f358015bd30 0x7f35800b1750 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f357000c750 tx=0x7f357000cb10 comp rx=0 tx=0).stop 2023-11-24T06:19:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 shutdown_connections 2023-11-24T06:19:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3580162920 0x7f35800af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f358015bd30 0x7f35800b1750 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f356c06af00 0x7f356c06d3c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 --2- 172.21.15.89:0/3773243273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358015e6e0 0x7f35800aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 >> 172.21.15.89:0/3773243273 conn(0x7f3580072710 msgr2=0x7f35800cccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 shutdown_connections 2023-11-24T06:19:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.596+0000 7f35852df700 1 -- 172.21.15.89:0/3773243273 wait complete. 2023-11-24T06:19:01.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:01.612 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:01.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.733+0000 7fc462eb6700 1 Processor -- start 2023-11-24T06:19:01.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.733+0000 7fc462eb6700 1 -- start start 2023-11-24T06:19:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.734+0000 7fc462eb6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c15e160 0x7fc45c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.734+0000 7fc462eb6700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0b3c60 con 0x7fc45c0afc90 2023-11-24T06:19:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.734+0000 7fc462eb6700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0b58c0 con 0x7fc45c0746a0 2023-11-24T06:19:01.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.734+0000 7fc462eb6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0b4500 con 0x7fc45c15e160 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.734+0000 7fc45bfff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc45c0afc90 0x7fc45c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39092/0 (socket says 172.21.15.89:39092) 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.734+0000 7fc45bfff700 1 -- 172.21.15.89:0/3232107689 learned_addr learned my addr 172.21.15.89:0/3232107689 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc460c52700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c15e160 0x7fc45c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957821134 0 0) 0x7fc45c0b3c60 con 0x7fc45c0afc90 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c0035b0 con 0x7fc45c0afc90 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015825681 0 0) 0x7fc45c0b58c0 con 0x7fc45c0746a0 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c0b3c60 con 0x7fc45c0746a0 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1027609498 0 0) 0x7fc44c0035b0 con 0x7fc45c0afc90 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c15e160 msgr2=0x7fc45c0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c15e160 0x7fc45c0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 >> v1:172.21.15.123:6789/0 conn(0x7fc45c0746a0 legacy=0x7fc45c06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c06bdc0 con 0x7fc45c0afc90 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fc448001670 con 0x7fc45c0afc90 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 >> v1:172.21.15.89:6789/0 conn(0x7fc45c0afc90 legacy=0x7fc45c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c003bc0 0x7fc44c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c004850 0x7fc44c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc44c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0b4500 con 0x7fc44c005210 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c00c000 con 0x7fc44c004850 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c00c1c0 con 0x7fc44c003bc0 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc460c52700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c15e160 0x7fc45c0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:01.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc45bfff700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c004850 0x7fc44c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc461453700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc44c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc460c52700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c003bc0 0x7fc44c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc461453700 1 -- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c003bc0 msgr2=0x7fc44c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc461453700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c003bc0 0x7fc44c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc461453700 1 -- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c004850 msgr2=0x7fc44c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc461453700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c004850 0x7fc44c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc461453700 1 -- 172.21.15.89:0/3232107689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c0035b0 con 0x7fc44c005210 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.735+0000 7fc460c52700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c003bc0 0x7fc44c003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc461453700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc44c00ba10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc454002230 tx=0x7fc44c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fd00cfe68f1392 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454001f30 con 0x7fc44c005210 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45400ce60 con 0x7fc44c005210 2023-11-24T06:19:01.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc45b7fe700 1 -- 172.21.15.89:0/3232107689 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454003c40 con 0x7fc44c005210 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc462eb6700 1 -- 172.21.15.89:0/3232107689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 msgr2=0x7fc44c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc462eb6700 1 --2- 172.21.15.89:0/3232107689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc44c00ba10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc454002230 tx=0x7fc44c005880 comp rx=0 tx=0).stop 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc461453700 1 -- 172.21.15.89:0/3232107689 reap_dead start 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc462eb6700 1 -- 172.21.15.89:0/3232107689 shutdown_connections 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc462eb6700 1 -- 172.21.15.89:0/3232107689 >> 172.21.15.89:0/3232107689 conn(0x7fc45c0d0bf0 msgr2=0x7fc45c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc462eb6700 1 -- 172.21.15.89:0/3232107689 shutdown_connections 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.736+0000 7fc462eb6700 1 -- 172.21.15.89:0/3232107689 wait complete. 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.737+0000 7fc462eb6700 1 Processor -- start 2023-11-24T06:19:01.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc462eb6700 1 -- start start 2023-11-24T06:19:01.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc462eb6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc45c155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc462eb6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc45c15f8d0 0x7fc45c157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc462eb6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c161290 0x7fc45c157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc462eb6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c160a50 con 0x7fc44c005210 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc462eb6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c15fdd0 con 0x7fc45c15f8d0 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc462eb6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c1600b0 con 0x7fc45c161290 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc460c52700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc45c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc460c52700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc45c155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35598/0 (socket says 172.21.15.89:35598) 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.738+0000 7fc460c52700 1 -- 172.21.15.89:0/2967413137 learned_addr learned my addr 172.21.15.89:0/2967413137 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc45bfff700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc45c15f8d0 0x7fc45c157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc461453700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c161290 0x7fc45c157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc460c52700 1 -- 172.21.15.89:0/2967413137 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c161290 msgr2=0x7fc45c157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc460c52700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c161290 0x7fc45c157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc460c52700 1 -- 172.21.15.89:0/2967413137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc45c15f8d0 msgr2=0x7fc45c157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc460c52700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc45c15f8d0 0x7fc45c157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc460c52700 1 -- 172.21.15.89:0/2967413137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c160390 con 0x7fc44c005210 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc460c52700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc45c155850 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4500118b0 tx=0x7fc450011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc4597fa700 1 -- 172.21.15.89:0/2967413137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450015870 con 0x7fc44c005210 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc4597fa700 1 -- 172.21.15.89:0/2967413137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4500159d0 con 0x7fc44c005210 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c160670 con 0x7fc44c005210 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc4597fa700 1 -- 172.21.15.89:0/2967413137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450015d40 con 0x7fc44c005210 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.739+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c160f20 con 0x7fc44c005210 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.740+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c0b54a0 con 0x7fc44c005210 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.740+0000 7fc4597fa700 1 -- 172.21.15.89:0/2967413137 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc45001d400 con 0x7fc44c005210 2023-11-24T06:19:01.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.741+0000 7fc4597fa700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc44006aed0 0x7fc44006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:01.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.741+0000 7fc45bfff700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc44006aed0 0x7fc44006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:01.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.741+0000 7fc4597fa700 1 -- 172.21.15.89:0/2967413137 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc450089d50 con 0x7fc44c005210 2023-11-24T06:19:01.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.741+0000 7fc45bfff700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc44006aed0 0x7fc44006d390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0d0490 tx=0x7fc448001e20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:01.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.745+0000 7fc4597fa700 1 -- 172.21.15.89:0/2967413137 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc45008d030 con 0x7fc44c005210 2023-11-24T06:19:01.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.937+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc45c068fb0 con 0x7fc44c005210 2023-11-24T06:19:01.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.937+0000 7fc4597fa700 1 -- 172.21.15.89:0/2967413137 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc450054e10 con 0x7fc44c005210 2023-11-24T06:19:01.940 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:01.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc44006aed0 msgr2=0x7fc44006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc44006aed0 0x7fc44006d390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0d0490 tx=0x7fc448001e20 comp rx=0 tx=0).stop 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 msgr2=0x7fc45c155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc45c155850 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4500118b0 tx=0x7fc450011bc0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 shutdown_connections 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc45c161290 0x7fc45c157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44c005210 0x7fc45c155850 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc44006aed0 0x7fc44006d390 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 --2- 172.21.15.89:0/2967413137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc45c15f8d0 0x7fc45c157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 >> 172.21.15.89:0/2967413137 conn(0x7fc45c0d0bf0 msgr2=0x7fc45c0aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 shutdown_connections 2023-11-24T06:19:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:01.939+0000 7fc462eb6700 1 -- 172.21.15.89:0/2967413137 wait complete. 2023-11-24T06:19:01.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.075+0000 7fb779f6c700 1 Processor -- start 2023-11-24T06:19:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.075+0000 7fb779f6c700 1 -- start start 2023-11-24T06:19:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb779f6c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb7740aedc0 0x7fb7740ae790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb779f6c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7740ad870 0x7fb77406f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb779f6c700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77406e400 con 0x7fb77415e290 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb779f6c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77406e630 con 0x7fb7740aedc0 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb779f6c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77406e910 con 0x7fb7740ad870 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb772ffd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb77415e290 0x7fb7740af1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39112/0 (socket says 172.21.15.89:39112) 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb772ffd700 1 -- 172.21.15.89:0/294399997 learned_addr learned my addr 172.21.15.89:0/294399997 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7737fe700 1 --2- 172.21.15.89:0/294399997 >> v2:172.21.15.123:3300/0 conn(0x7fb7740aedc0 0x7fb7740ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080285677 0 0) 0x7fb77406e400 con 0x7fb77415e290 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb773fff700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7740ad870 0x7fb77406f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7600035b0 con 0x7fb77415e290 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710381282 0 0) 0x7fb7600035b0 con 0x7fb77415e290 2023-11-24T06:19:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7740ad870 msgr2=0x7fb77406f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7740ad870 0x7fb77406f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 >> v2:172.21.15.123:3300/0 conn(0x7fb7740aedc0 msgr2=0x7fb7740ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 --2- 172.21.15.89:0/294399997 >> v2:172.21.15.123:3300/0 conn(0x7fb7740aedc0 0x7fb7740ae790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77406ebf0 con 0x7fb77415e290 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.076+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fb764001670 con 0x7fb77415e290 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 >> v1:172.21.15.89:6789/0 conn(0x7fb77415e290 legacy=0x7fb7740af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb760003a00 0x7fb760003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760004850 0x7fb760004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb760005210 0x7fb76000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77406e630 con 0x7fb760003a00 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77406e910 con 0x7fb760004850 2023-11-24T06:19:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76000c030 con 0x7fb760005210 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb773fff700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7740ad870 0x7fb77406f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7737fe700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb760003a00 0x7fb760003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb772ffd700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760004850 0x7fb760004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb773fff700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb760005210 0x7fb76000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7737fe700 1 -- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb760005210 msgr2=0x7fb76000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7737fe700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb760005210 0x7fb76000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7737fe700 1 -- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760004850 msgr2=0x7fb760004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7737fe700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760004850 0x7fb760004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7737fe700 1 -- 172.21.15.89:0/294399997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7600035b0 con 0x7fb760003a00 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb772ffd700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760004850 0x7fb760004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7737fe700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb760003a00 0x7fb760003e20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009b60 tx=0x7fb760005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff04b23d1eadf79 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00e6c0 con 0x7fb760003a00 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c00e820 con 0x7fb760003a00 2023-11-24T06:19:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.077+0000 7fb7727fc700 1 -- 172.21.15.89:0/294399997 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00eb90 con 0x7fb760003a00 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.078+0000 7fb779f6c700 1 -- 172.21.15.89:0/294399997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb760003a00 msgr2=0x7fb760003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.078+0000 7fb779f6c700 1 --2- 172.21.15.89:0/294399997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb760003a00 0x7fb760003e20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009b60 tx=0x7fb760005880 comp rx=0 tx=0).stop 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.078+0000 7fb773fff700 1 -- 172.21.15.89:0/294399997 reap_dead start 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.078+0000 7fb779f6c700 1 -- 172.21.15.89:0/294399997 shutdown_connections 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.078+0000 7fb779f6c700 1 -- 172.21.15.89:0/294399997 >> 172.21.15.89:0/294399997 conn(0x7fb7740b2050 msgr2=0x7fb7740b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.078+0000 7fb779f6c700 1 -- 172.21.15.89:0/294399997 shutdown_connections 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.078+0000 7fb779f6c700 1 -- 172.21.15.89:0/294399997 wait complete. 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.079+0000 7fb779f6c700 1 Processor -- start 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.079+0000 7fb779f6c700 1 -- start start 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb779f6c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760003a00 0x7fb7740c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb779f6c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb77415a6e0 0x7fb77415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb779f6c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb774147580 0x7fb774149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb779f6c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77415d5e0 con 0x7fb774147580 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb779f6c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb774148000 con 0x7fb760003a00 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb779f6c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7741482e0 con 0x7fb77415a6e0 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb774147580 0x7fb774149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb774147580 0x7fb774149ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35622/0 (socket says 172.21.15.89:35622) 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 -- 172.21.15.89:0/1786269372 learned_addr learned my addr 172.21.15.89:0/1786269372 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb772ffd700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb77415a6e0 0x7fb77415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb7737fe700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760003a00 0x7fb7740c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 -- 172.21.15.89:0/1786269372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb77415a6e0 msgr2=0x7fb77415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb77415a6e0 0x7fb77415d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 -- 172.21.15.89:0/1786269372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760003a00 msgr2=0x7fb7740c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760003a00 0x7fb7740c53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 -- 172.21.15.89:0/1786269372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7741485c0 con 0x7fb774147580 2023-11-24T06:19:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb773fff700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb774147580 0x7fb774149ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7740724d0 tx=0x7fb76c0048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb76bfff700 1 -- 172.21.15.89:0/1786269372 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c004930 con 0x7fb774147580 2023-11-24T06:19:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb76bfff700 1 -- 172.21.15.89:0/1786269372 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c00f050 con 0x7fb774147580 2023-11-24T06:19:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb76bfff700 1 -- 172.21.15.89:0/1786269372 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c007da0 con 0x7fb774147580 2023-11-24T06:19:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.080+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7741488a0 con 0x7fb774147580 2023-11-24T06:19:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.081+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77415ab60 con 0x7fb774147580 2023-11-24T06:19:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.082+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb774148ef0 con 0x7fb774147580 2023-11-24T06:19:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.082+0000 7fb76bfff700 1 -- 172.21.15.89:0/1786269372 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb76c01e460 con 0x7fb774147580 2023-11-24T06:19:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.082+0000 7fb76bfff700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75406aed0 0x7fb75406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.082+0000 7fb76bfff700 1 -- 172.21.15.89:0/1786269372 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb76c089930 con 0x7fb774147580 2023-11-24T06:19:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.082+0000 7fb7737fe700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75406aed0 0x7fb75406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.083+0000 7fb7737fe700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75406aed0 0x7fb75406d390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0090b0 tx=0x7fb75c01f040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.086+0000 7fb76bfff700 1 -- 172.21.15.89:0/1786269372 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb76c0549e0 con 0x7fb774147580 2023-11-24T06:19:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.279+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb774068d90 con 0x7fb774147580 2023-11-24T06:19:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.280+0000 7fb76bfff700 1 -- 172.21.15.89:0/1786269372 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb76c058000 con 0x7fb774147580 2023-11-24T06:19:02.283 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.281+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75406aed0 msgr2=0x7fb75406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.281+0000 7fb779f6c700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75406aed0 0x7fb75406d390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0090b0 tx=0x7fb75c01f040 comp rx=0 tx=0).stop 2023-11-24T06:19:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.281+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb774147580 msgr2=0x7fb774149ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.281+0000 7fb779f6c700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb774147580 0x7fb774149ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7740724d0 tx=0x7fb76c0048c0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.282+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 shutdown_connections 2023-11-24T06:19:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.282+0000 7fb779f6c700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb774147580 0x7fb774149ef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.282+0000 7fb779f6c700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75406aed0 0x7fb75406d390 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.282+0000 7fb779f6c700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb760003a00 0x7fb7740c53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.282+0000 7fb779f6c700 1 --2- 172.21.15.89:0/1786269372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb77415a6e0 0x7fb77415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.282+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 >> 172.21.15.89:0/1786269372 conn(0x7fb7740b2050 msgr2=0x7fb7740b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.282+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 shutdown_connections 2023-11-24T06:19:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.282+0000 7fb779f6c700 1 -- 172.21.15.89:0/1786269372 wait complete. 2023-11-24T06:19:02.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:02.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.417+0000 7f6ea79cf700 1 Processor -- start 2023-11-24T06:19:02.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.418+0000 7f6ea79cf700 1 -- start start 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.418+0000 7f6ea79cf700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea015e160 0x7f6ea00b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.418+0000 7f6ea79cf700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ea00746a0 0x7f6ea006d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.418+0000 7f6ea79cf700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea00b3c60 con 0x7f6ea00afc90 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.418+0000 7f6ea79cf700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea00b4500 con 0x7f6ea00746a0 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.418+0000 7f6ea79cf700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00b58c0 con 0x7f6ea015e160 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.418+0000 7f6ea5f6c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ea00746a0 0x7f6ea006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6ea5f6c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ea00746a0 0x7f6ea006d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35630/0 (socket says 172.21.15.89:35630) 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6ea5f6c700 1 -- 172.21.15.89:0/2150705425 learned_addr learned my addr 172.21.15.89:0/2150705425 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6ea576b700 1 --2- 172.21.15.89:0/2150705425 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea015e160 0x7f6ea00b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6ea5f6c700 1 -- 172.21.15.89:0/2150705425 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea015e160 msgr2=0x7f6ea00b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6ea5f6c700 1 --2- 172.21.15.89:0/2150705425 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea015e160 0x7f6ea00b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6ea5f6c700 1 -- 172.21.15.89:0/2150705425 >> v1:172.21.15.123:6789/0 conn(0x7f6ea00afc90 legacy=0x7f6ea0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6ea5f6c700 1 -- 172.21.15.89:0/2150705425 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea006bdc0 con 0x7f6ea00746a0 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6ea5f6c700 1 --2- 172.21.15.89:0/2150705425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea00746a0 0x7f6ea006d930 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009e50 tx=0x7f6e9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fd4cf2a19b00be server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6e97fff700 1 -- 172.21.15.89:0/2150705425 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800e070 con 0x7f6ea00746a0 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6e97fff700 1 -- 172.21.15.89:0/2150705425 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9800b9c0 con 0x7f6ea00746a0 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.419+0000 7f6e97fff700 1 -- 172.21.15.89:0/2150705425 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800bd70 con 0x7f6ea00746a0 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.420+0000 7f6ea79cf700 1 -- 172.21.15.89:0/2150705425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea00746a0 msgr2=0x7f6ea006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.420+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/2150705425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea00746a0 0x7f6ea006d930 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009e50 tx=0x7f6e9800b330 comp rx=0 tx=0).stop 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.420+0000 7f6ea79cf700 1 -- 172.21.15.89:0/2150705425 shutdown_connections 2023-11-24T06:19:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.420+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/2150705425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea00746a0 0x7f6ea006d930 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.420+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/2150705425 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea015e160 0x7f6ea00b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.420+0000 7f6ea79cf700 1 -- 172.21.15.89:0/2150705425 >> 172.21.15.89:0/2150705425 conn(0x7f6ea00d0bf0 msgr2=0x7f6ea00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:02.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.420+0000 7f6ea79cf700 1 -- 172.21.15.89:0/2150705425 shutdown_connections 2023-11-24T06:19:02.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.420+0000 7f6ea79cf700 1 -- 172.21.15.89:0/2150705425 wait complete. 2023-11-24T06:19:02.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.421+0000 7f6ea79cf700 1 Processor -- start 2023-11-24T06:19:02.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.421+0000 7f6ea79cf700 1 -- start start 2023-11-24T06:19:02.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea79cf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea00746a0 0x7f6ea0158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea79cf700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea00afc90 0x7f6ea0160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea79cf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea015e160 0x7f6ea0160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea79cf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea0160200 con 0x7f6ea015e160 2023-11-24T06:19:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea79cf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea015f5f0 con 0x7f6ea00746a0 2023-11-24T06:19:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea79cf700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea015f8d0 con 0x7f6ea00afc90 2023-11-24T06:19:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea015e160 0x7f6ea0160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea015e160 0x7f6ea0160fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35636/0 (socket says 172.21.15.89:35636) 2023-11-24T06:19:02.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 -- 172.21.15.89:0/675369900 learned_addr learned my addr 172.21.15.89:0/675369900 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:02.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea576b700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea00746a0 0x7f6ea0158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea4f6a700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea00afc90 0x7f6ea0160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 -- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea00afc90 msgr2=0x7f6ea0160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea00afc90 0x7f6ea0160a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 -- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea00746a0 msgr2=0x7f6ea0158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea00746a0 0x7f6ea0158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 -- 172.21.15.89:0/675369900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98009b30 con 0x7f6ea015e160 2023-11-24T06:19:02.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea576b700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea00746a0 0x7f6ea0158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:02.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea5f6c700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea015e160 0x7f6ea0160fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6e98012000 tx=0x7f6e98004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6e95ffb700 1 -- 172.21.15.89:0/675369900 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98004920 con 0x7f6ea015e160 2023-11-24T06:19:02.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6e95ffb700 1 -- 172.21.15.89:0/675369900 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e980099b0 con 0x7f6ea015e160 2023-11-24T06:19:02.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea015fbb0 con 0x7f6ea015e160 2023-11-24T06:19:02.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.422+0000 7f6e95ffb700 1 -- 172.21.15.89:0/675369900 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9801c680 con 0x7f6ea015e160 2023-11-24T06:19:02.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.423+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea0158dd0 con 0x7f6ea015e160 2023-11-24T06:19:02.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.423+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea0074e60 con 0x7f6ea015e160 2023-11-24T06:19:02.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.424+0000 7f6e95ffb700 1 -- 172.21.15.89:0/675369900 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6e9801c7e0 con 0x7f6ea015e160 2023-11-24T06:19:02.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.425+0000 7f6e95ffb700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e8406aed0 0x7f6e8406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.425+0000 7f6e95ffb700 1 -- 172.21.15.89:0/675369900 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e9808b910 con 0x7f6ea015e160 2023-11-24T06:19:02.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.425+0000 7f6ea576b700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e8406aed0 0x7f6e8406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.425+0000 7f6ea576b700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e8406aed0 0x7f6e8406d390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009820 tx=0x7f6e90008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.429+0000 7f6e95ffb700 1 -- 172.21.15.89:0/675369900 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e98056a30 con 0x7f6ea015e160 2023-11-24T06:19:02.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.619+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6ea0068fb0 con 0x7f6ea015e160 2023-11-24T06:19:02.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.620+0000 7f6e95ffb700 1 -- 172.21.15.89:0/675369900 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6e9805a050 con 0x7f6ea015e160 2023-11-24T06:19:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:02.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e8406aed0 msgr2=0x7f6e8406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e8406aed0 0x7f6e8406d390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009820 tx=0x7f6e90008040 comp rx=0 tx=0).stop 2023-11-24T06:19:02.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea015e160 msgr2=0x7f6ea0160fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea015e160 0x7f6ea0160fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6e98012000 tx=0x7f6e98004ad0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 shutdown_connections 2023-11-24T06:19:02.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ea015e160 0x7f6ea0160fa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e8406aed0 0x7f6e8406d390 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea00746a0 0x7f6ea0158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 --2- 172.21.15.89:0/675369900 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ea00afc90 0x7f6ea0160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 >> 172.21.15.89:0/675369900 conn(0x7f6ea00d0bf0 msgr2=0x7f6ea00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:02.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 shutdown_connections 2023-11-24T06:19:02.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.622+0000 7f6ea79cf700 1 -- 172.21.15.89:0/675369900 wait complete. 2023-11-24T06:19:02.637 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:02.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.754+0000 7f0973d1a700 1 Processor -- start 2023-11-24T06:19:02.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.755+0000 7f0973d1a700 1 -- start start 2023-11-24T06:19:02.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.755+0000 7f0973d1a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f096c15b200 0x7f096c0b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.755+0000 7f0973d1a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c0b1a70 0x7f096c164b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.755+0000 7f0973d1a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096c0cfd40 con 0x7f096c15ae20 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.755+0000 7f0973d1a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f096c0d0020 con 0x7f096c15b200 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.755+0000 7f0973d1a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096c0affa0 con 0x7f096c0b1a70 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09722b7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c0b1a70 0x7f096c164b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09712b5700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f096c15b200 0x7f096c0b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09712b5700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f096c15b200 0x7f096c0b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:50754/0 (socket says 172.21.15.89:50754) 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09712b5700 1 -- 172.21.15.89:0/3487185793 learned_addr learned my addr 172.21.15.89:0/3487185793 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f0971ab6700 1 --1- 172.21.15.89:0/3487185793 >> v1:172.21.15.123:6789/0 conn(0x7f096c15ae20 0x7f096c0b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:42966/0 (socket says 172.21.15.89:42966) 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09712b5700 1 -- 172.21.15.89:0/3487185793 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c0b1a70 msgr2=0x7f096c164b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09712b5700 1 --2- 172.21.15.89:0/3487185793 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c0b1a70 0x7f096c164b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09712b5700 1 -- 172.21.15.89:0/3487185793 >> v1:172.21.15.123:6789/0 conn(0x7f096c15ae20 legacy=0x7f096c0b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09712b5700 1 -- 172.21.15.89:0/3487185793 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c0b0280 con 0x7f096c15b200 2023-11-24T06:19:02.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f0970ab4700 1 -- 172.21.15.89:0/3487185793 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171835160 0 0) 0x7f096c0cfd40 con 0x7f096c15ae20 2023-11-24T06:19:02.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f09712b5700 1 --2- 172.21.15.89:0/3487185793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 0x7f096c0b29a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f09680098f0 tx=0x7f0968009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b187cab3ce0de0e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f0970ab4700 1 -- 172.21.15.89:0/3487185793 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968011830 con 0x7f096c15b200 2023-11-24T06:19:02.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f0970ab4700 1 -- 172.21.15.89:0/3487185793 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0968011990 con 0x7f096c15b200 2023-11-24T06:19:02.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.756+0000 7f0970ab4700 1 -- 172.21.15.89:0/3487185793 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968011d20 con 0x7f096c15b200 2023-11-24T06:19:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.757+0000 7f0973d1a700 1 -- 172.21.15.89:0/3487185793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 msgr2=0x7f096c0b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.757+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3487185793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 0x7f096c0b29a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f09680098f0 tx=0x7f0968009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.757+0000 7f0973d1a700 1 -- 172.21.15.89:0/3487185793 shutdown_connections 2023-11-24T06:19:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.757+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3487185793 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c0b1a70 0x7f096c164b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.757+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3487185793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 0x7f096c0b29a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.757+0000 7f0973d1a700 1 -- 172.21.15.89:0/3487185793 >> 172.21.15.89:0/3487185793 conn(0x7f096c0b57b0 msgr2=0x7f096c0b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.757+0000 7f0973d1a700 1 -- 172.21.15.89:0/3487185793 shutdown_connections 2023-11-24T06:19:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.757+0000 7f0973d1a700 1 -- 172.21.15.89:0/3487185793 wait complete. 2023-11-24T06:19:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.758+0000 7f0973d1a700 1 Processor -- start 2023-11-24T06:19:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.758+0000 7f0973d1a700 1 -- start start 2023-11-24T06:19:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0973d1a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f096c0b1a70 0x7f096c159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0973d1a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c15ae20 0x7f096c156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0973d1a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 0x7f096c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0973d1a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c0cc4c0 con 0x7f096c0b1a70 2023-11-24T06:19:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0973d1a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c0cb8b0 con 0x7f096c15b200 2023-11-24T06:19:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0973d1a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096c0cbb90 con 0x7f096c15ae20 2023-11-24T06:19:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0971ab6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f096c0b1a70 0x7f096c159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0971ab6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f096c0b1a70 0x7f096c159e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35646/0 (socket says 172.21.15.89:35646) 2023-11-24T06:19:02.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0971ab6700 1 -- 172.21.15.89:0/3546110028 learned_addr learned my addr 172.21.15.89:0/3546110028 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:02.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f09722b7700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 0x7f096c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f09712b5700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c15ae20 0x7f096c156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0971ab6700 1 -- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c15ae20 msgr2=0x7f096c156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0971ab6700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c15ae20 0x7f096c156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0971ab6700 1 -- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 msgr2=0x7f096c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0971ab6700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 0x7f096c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f0971ab6700 1 -- 172.21.15.89:0/3546110028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0968009550 con 0x7f096c0b1a70 2023-11-24T06:19:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.759+0000 7f09722b7700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 0x7f096c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.760+0000 7f0971ab6700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f096c0b1a70 0x7f096c159e50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f095c00c520 tx=0x7f095c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.760+0000 7f09627fc700 1 -- 172.21.15.89:0/3546110028 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c004c80 con 0x7f096c0b1a70 2023-11-24T06:19:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.760+0000 7f09627fc700 1 -- 172.21.15.89:0/3546110028 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095c004e00 con 0x7f096c0b1a70 2023-11-24T06:19:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.760+0000 7f09627fc700 1 -- 172.21.15.89:0/3546110028 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c007310 con 0x7f096c0b1a70 2023-11-24T06:19:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.760+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c0cbe70 con 0x7f096c0b1a70 2023-11-24T06:19:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.760+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c154b70 con 0x7f096c0b1a70 2023-11-24T06:19:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.761+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c067f60 con 0x7f096c0b1a70 2023-11-24T06:19:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.761+0000 7f09627fc700 1 -- 172.21.15.89:0/3546110028 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f095c007470 con 0x7f096c0b1a70 2023-11-24T06:19:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.761+0000 7f09627fc700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f095006af00 0x7f095006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.761+0000 7f09627fc700 1 -- 172.21.15.89:0/3546110028 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f095c0893b0 con 0x7f096c0b1a70 2023-11-24T06:19:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.762+0000 7f09712b5700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f095006af00 0x7f095006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.762+0000 7f09712b5700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f095006af00 0x7f095006d3c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f09680098f0 tx=0x7f0968008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:02.766+0000 7f09627fc700 1 -- 172.21.15.89:0/3546110028 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f095c0543d0 con 0x7f096c0b1a70 2023-11-24T06:19:03.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.000+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f096c0ca250 con 0x7f096c0b1a70 2023-11-24T06:19:03.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.001+0000 7f09627fc700 1 -- 172.21.15.89:0/3546110028 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f095c0579f0 con 0x7f096c0b1a70 2023-11-24T06:19:03.004 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:03.004 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:18:49.391199+0000","modified":"2023-11-24T06:18:52.422301+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:19:03.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f095006af00 msgr2=0x7f095006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f095006af00 0x7f095006d3c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f09680098f0 tx=0x7f0968008030 comp rx=0 tx=0).stop 2023-11-24T06:19:03.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f096c0b1a70 msgr2=0x7f096c159e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f096c0b1a70 0x7f096c159e50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f095c00c520 tx=0x7f095c00c8e0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 shutdown_connections 2023-11-24T06:19:03.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f095006af00 0x7f095006d3c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f096c15b200 0x7f096c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f096c0b1a70 0x7f096c159e50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 --2- 172.21.15.89:0/3546110028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f096c15ae20 0x7f096c156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 >> 172.21.15.89:0/3546110028 conn(0x7f096c0b57b0 msgr2=0x7f096c0b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:03.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 shutdown_connections 2023-11-24T06:19:03.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.003+0000 7f0973d1a700 1 -- 172.21.15.89:0/3546110028 wait complete. 2023-11-24T06:19:03.008 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-11-24T06:19:03.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.135+0000 7fca38799700 1 Processor -- start 2023-11-24T06:19:03.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.135+0000 7fca38799700 1 -- start start 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.135+0000 7fca38799700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fca300c9fc0 0x7fca301691b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.135+0000 7fca38799700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca300cfff0 con 0x7fca30158670 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.135+0000 7fca38799700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca300cfd10 con 0x7fca30158a50 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.135+0000 7fca38799700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca300629b0 con 0x7fca300c9fc0 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca36d36700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fca300c9fc0 0x7fca301691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca36d36700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fca300c9fc0 0x7fca301691b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35658/0 (socket says 172.21.15.89:35658) 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca36535700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fca30158670 0x7fca300622a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39158/0 (socket says 172.21.15.89:39158) 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca36d36700 1 -- 172.21.15.89:0/2801776675 learned_addr learned my addr 172.21.15.89:0/2801776675 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca36d36700 1 -- 172.21.15.89:0/2801776675 >> v1:172.21.15.123:6789/0 conn(0x7fca30158a50 legacy=0x7fca300c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca36d36700 1 -- 172.21.15.89:0/2801776675 >> v1:172.21.15.89:6789/0 conn(0x7fca30158670 legacy=0x7fca300622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca36d36700 1 -- 172.21.15.89:0/2801776675 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30161f50 con 0x7fca300c9fc0 2023-11-24T06:19:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca35533700 1 -- 172.21.15.89:0/2801776675 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260439195 0 0) 0x7fca300cfff0 con 0x7fca30158670 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca36d36700 1 --2- 172.21.15.89:0/2801776675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca300c9fc0 0x7fca301691b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fca28009e50 tx=0x7fca2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbaaf9883d1fc8a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca35533700 1 -- 172.21.15.89:0/2801776675 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2800e070 con 0x7fca300c9fc0 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca35533700 1 -- 172.21.15.89:0/2801776675 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2800b9c0 con 0x7fca300c9fc0 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.136+0000 7fca35533700 1 -- 172.21.15.89:0/2801776675 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2800bd70 con 0x7fca300c9fc0 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.137+0000 7fca38799700 1 -- 172.21.15.89:0/2801776675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca300c9fc0 msgr2=0x7fca301691b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.137+0000 7fca38799700 1 --2- 172.21.15.89:0/2801776675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca300c9fc0 0x7fca301691b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fca28009e50 tx=0x7fca2800b330 comp rx=0 tx=0).stop 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.137+0000 7fca38799700 1 -- 172.21.15.89:0/2801776675 shutdown_connections 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.137+0000 7fca38799700 1 --2- 172.21.15.89:0/2801776675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca300c9fc0 0x7fca301691b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.137+0000 7fca38799700 1 -- 172.21.15.89:0/2801776675 >> 172.21.15.89:0/2801776675 conn(0x7fca300b57b0 msgr2=0x7fca300b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.137+0000 7fca38799700 1 -- 172.21.15.89:0/2801776675 shutdown_connections 2023-11-24T06:19:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.137+0000 7fca38799700 1 -- 172.21.15.89:0/2801776675 wait complete. 2023-11-24T06:19:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.138+0000 7fca38799700 1 Processor -- start 2023-11-24T06:19:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.138+0000 7fca38799700 1 -- start start 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca38799700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca300c9fc0 0x7fca3015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca38799700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca30158670 0x7fca300cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca38799700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca30158a50 0x7fca30159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca38799700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca300b45d0 con 0x7fca30158a50 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca38799700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca300b3cc0 con 0x7fca30158670 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca38799700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3015bb30 con 0x7fca300c9fc0 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca30158a50 0x7fca30159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca30158a50 0x7fca30159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35664/0 (socket says 172.21.15.89:35664) 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 -- 172.21.15.89:0/2024361199 learned_addr learned my addr 172.21.15.89:0/2024361199 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca35d34700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca30158670 0x7fca300cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36535700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca300c9fc0 0x7fca3015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 -- 172.21.15.89:0/2024361199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca300c9fc0 msgr2=0x7fca3015b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca300c9fc0 0x7fca3015b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 -- 172.21.15.89:0/2024361199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca30158670 msgr2=0x7fca300cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca30158670 0x7fca300cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 -- 172.21.15.89:0/2024361199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca28009b30 con 0x7fca30158a50 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca36d36700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca30158a50 0x7fca30159970 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fca2800b730 tx=0x7fca28004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca26ffd700 1 -- 172.21.15.89:0/2024361199 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2801c640 con 0x7fca30158a50 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca26ffd700 1 -- 172.21.15.89:0/2024361199 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2801c7a0 con 0x7fca30158a50 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca26ffd700 1 -- 172.21.15.89:0/2024361199 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2801cb50 con 0x7fca30158a50 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca300b3f10 con 0x7fca30158a50 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.139+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca300cb160 con 0x7fca30158a50 2023-11-24T06:19:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.140+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca300b4710 con 0x7fca30158a50 2023-11-24T06:19:03.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.141+0000 7fca26ffd700 1 -- 172.21.15.89:0/2024361199 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fca2801ccb0 con 0x7fca30158a50 2023-11-24T06:19:03.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.141+0000 7fca26ffd700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fca1406aed0 0x7fca1406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.141+0000 7fca36535700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fca1406aed0 0x7fca1406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.141+0000 7fca26ffd700 1 -- 172.21.15.89:0/2024361199 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca2808afa0 con 0x7fca30158a50 2023-11-24T06:19:03.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.142+0000 7fca36535700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fca1406aed0 0x7fca1406d390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fca20007fd0 tx=0x7fca20007c20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:03.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.146+0000 7fca26ffd700 1 -- 172.21.15.89:0/2024361199 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca280bde90 con 0x7fca30158a50 2023-11-24T06:19:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.336+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca30067f60 con 0x7fca30158a50 2023-11-24T06:19:03.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.337+0000 7fca26ffd700 1 -- 172.21.15.89:0/2024361199 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7fca300b4710 con 0x7fca30158a50 2023-11-24T06:19:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:03.341 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:03.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.339+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fca1406aed0 msgr2=0x7fca1406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.339+0000 7fca38799700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fca1406aed0 0x7fca1406d390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fca20007fd0 tx=0x7fca20007c20 comp rx=0 tx=0).stop 2023-11-24T06:19:03.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.339+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca30158a50 msgr2=0x7fca30159970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.339+0000 7fca38799700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca30158a50 0x7fca30159970 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fca2800b730 tx=0x7fca28004ad0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.340+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 shutdown_connections 2023-11-24T06:19:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.340+0000 7fca38799700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca30158a50 0x7fca30159970 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.340+0000 7fca38799700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca300c9fc0 0x7fca3015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.340+0000 7fca38799700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fca1406aed0 0x7fca1406d390 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.340+0000 7fca38799700 1 --2- 172.21.15.89:0/2024361199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca30158670 0x7fca300cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.340+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 >> 172.21.15.89:0/2024361199 conn(0x7fca300b57b0 msgr2=0x7fca3006e160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.340+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 shutdown_connections 2023-11-24T06:19:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.340+0000 7fca38799700 1 -- 172.21.15.89:0/2024361199 wait complete. 2023-11-24T06:19:03.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:03.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.473+0000 7f12ebd67700 1 Processor -- start 2023-11-24T06:19:03.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.473+0000 7f12ebd67700 1 -- start start 2023-11-24T06:19:03.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12ebd67700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f12e415e080 0x7f12e40b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12ebd67700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e40af4d0 0x7f12e40afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12ebd67700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e40ae430 con 0x7f12e406c000 2023-11-24T06:19:03.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12ebd67700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e40af0b0 con 0x7f12e415e080 2023-11-24T06:19:03.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12ebd67700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e40b05a0 con 0x7f12e40af4d0 2023-11-24T06:19:03.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12e9b03700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f12e415e080 0x7f12e40b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12ea304700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f12e406c000 0x7f12e4073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39174/0 (socket says 172.21.15.89:39174) 2023-11-24T06:19:03.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12e9b03700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f12e415e080 0x7f12e40b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35668/0 (socket says 172.21.15.89:35668) 2023-11-24T06:19:03.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12ea304700 1 -- 172.21.15.89:0/2459343852 learned_addr learned my addr 172.21.15.89:0/2459343852 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:03.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12e9302700 1 --2- 172.21.15.89:0/2459343852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e40af4d0 0x7f12e40afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12e9b03700 1 -- 172.21.15.89:0/2459343852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e40af4d0 msgr2=0x7f12e40afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:03.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12e9b03700 1 --2- 172.21.15.89:0/2459343852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e40af4d0 0x7f12e40afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.474+0000 7f12e9b03700 1 -- 172.21.15.89:0/2459343852 >> v1:172.21.15.89:6789/0 conn(0x7f12e406c000 legacy=0x7f12e4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12e8b01700 1 -- 172.21.15.89:0/2459343852 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43981180 0 0) 0x7f12e40ae430 con 0x7f12e406c000 2023-11-24T06:19:03.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12e9b03700 1 -- 172.21.15.89:0/2459343852 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e40ae710 con 0x7f12e415e080 2023-11-24T06:19:03.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12e9302700 1 --2- 172.21.15.89:0/2459343852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e40af4d0 0x7f12e40afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:03.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12e9b03700 1 --2- 172.21.15.89:0/2459343852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e415e080 0x7f12e40b5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009870 tx=0x7f12d4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bead1b51b308b32 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:03.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12e8b01700 1 -- 172.21.15.89:0/2459343852 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d4011830 con 0x7f12e415e080 2023-11-24T06:19:03.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12e8b01700 1 -- 172.21.15.89:0/2459343852 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d4011990 con 0x7f12e415e080 2023-11-24T06:19:03.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12e8b01700 1 -- 172.21.15.89:0/2459343852 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d4011d20 con 0x7f12e415e080 2023-11-24T06:19:03.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12ebd67700 1 -- 172.21.15.89:0/2459343852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e415e080 msgr2=0x7f12e40b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2459343852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e415e080 0x7f12e40b5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009870 tx=0x7f12d4009b80 comp rx=0 tx=0).stop 2023-11-24T06:19:03.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12ebd67700 1 -- 172.21.15.89:0/2459343852 shutdown_connections 2023-11-24T06:19:03.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2459343852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e415e080 0x7f12e40b5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2459343852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e40af4d0 0x7f12e40afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12ebd67700 1 -- 172.21.15.89:0/2459343852 >> 172.21.15.89:0/2459343852 conn(0x7f12e4071b90 msgr2=0x7f12e4071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:03.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12ebd67700 1 -- 172.21.15.89:0/2459343852 shutdown_connections 2023-11-24T06:19:03.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.475+0000 7f12ebd67700 1 -- 172.21.15.89:0/2459343852 wait complete. 2023-11-24T06:19:03.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12ebd67700 1 Processor -- start 2023-11-24T06:19:03.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12ebd67700 1 -- start start 2023-11-24T06:19:03.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12ebd67700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e406c000 0x7f12e41f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12ebd67700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e40af4d0 0x7f12e4157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12ebd67700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e415e080 0x7f12e40cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12ebd67700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e415a890 con 0x7f12e40af4d0 2023-11-24T06:19:03.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12ebd67700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e41f3570 con 0x7f12e415e080 2023-11-24T06:19:03.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12ebd67700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e41580a0 con 0x7f12e406c000 2023-11-24T06:19:03.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12e9302700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e40af4d0 0x7f12e4157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12e9302700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e40af4d0 0x7f12e4157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35674/0 (socket says 172.21.15.89:35674) 2023-11-24T06:19:03.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12e9302700 1 -- 172.21.15.89:0/2411153608 learned_addr learned my addr 172.21.15.89:0/2411153608 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:03.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.477+0000 7f12e9b03700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e406c000 0x7f12e41f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12ea304700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e415e080 0x7f12e40cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12e9302700 1 -- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e406c000 msgr2=0x7f12e41f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12e9302700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e406c000 0x7f12e41f3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12e9302700 1 -- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e415e080 msgr2=0x7f12e40cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12e9302700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e415e080 0x7f12e40cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12e9302700 1 -- 172.21.15.89:0/2411153608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d40094a0 con 0x7f12e40af4d0 2023-11-24T06:19:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12ea304700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e415e080 0x7f12e40cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12e9b03700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e406c000 0x7f12e41f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12e9302700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e40af4d0 0x7f12e4157b60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f12e000c740 tx=0x7f12e000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12da7fc700 1 -- 172.21.15.89:0/2411153608 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0004220 con 0x7f12e40af4d0 2023-11-24T06:19:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12da7fc700 1 -- 172.21.15.89:0/2411153608 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e0004380 con 0x7f12e40af4d0 2023-11-24T06:19:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e4158370 con 0x7f12e40af4d0 2023-11-24T06:19:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12da7fc700 1 -- 172.21.15.89:0/2411153608 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e00036a0 con 0x7f12e40af4d0 2023-11-24T06:19:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.478+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e4158930 con 0x7f12e40af4d0 2023-11-24T06:19:03.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.479+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e40753f0 con 0x7f12e40af4d0 2023-11-24T06:19:03.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.479+0000 7f12da7fc700 1 -- 172.21.15.89:0/2411153608 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f12e0003800 con 0x7f12e40af4d0 2023-11-24T06:19:03.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.480+0000 7f12da7fc700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f12c806aed0 0x7f12c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.480+0000 7f12da7fc700 1 -- 172.21.15.89:0/2411153608 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12e00898d0 con 0x7f12e40af4d0 2023-11-24T06:19:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.480+0000 7f12e9b03700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f12c806aed0 0x7f12c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.480+0000 7f12e9b03700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f12c806aed0 0x7f12c806d390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f12d4000c00 tx=0x7f12d401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.484+0000 7f12da7fc700 1 -- 172.21.15.89:0/2411153608 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12e0054990 con 0x7f12e40af4d0 2023-11-24T06:19:03.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.723+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e4067f60 con 0x7f12e40af4d0 2023-11-24T06:19:03.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.723+0000 7f12da7fc700 1 -- 172.21.15.89:0/2411153608 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f12e0057fb0 con 0x7f12e40af4d0 2023-11-24T06:19:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:03.727 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:18:49.391199+0000","modified":"2023-11-24T06:18:52.422301+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:19:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.725+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f12c806aed0 msgr2=0x7f12c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.725+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f12c806aed0 0x7f12c806d390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f12d4000c00 tx=0x7f12d401c040 comp rx=0 tx=0).stop 2023-11-24T06:19:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.725+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e40af4d0 msgr2=0x7f12e4157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.725+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e40af4d0 0x7f12e4157b60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f12e000c740 tx=0x7f12e000cb00 comp rx=0 tx=0).stop 2023-11-24T06:19:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.726+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 shutdown_connections 2023-11-24T06:19:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.726+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f12c806aed0 0x7f12c806d390 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.726+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e415e080 0x7f12e40cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.726+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12e406c000 0x7f12e41f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.726+0000 7f12ebd67700 1 --2- 172.21.15.89:0/2411153608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e40af4d0 0x7f12e4157b60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.726+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 >> 172.21.15.89:0/2411153608 conn(0x7f12e4071b90 msgr2=0x7f12e406f300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:03.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.726+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 shutdown_connections 2023-11-24T06:19:03.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:03.726+0000 7f12ebd67700 1 -- 172.21.15.89:0/2411153608 wait complete. 2023-11-24T06:19:03.731 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-11-24T06:19:04.143 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T06:18:49.391199+0000', 'modified': '2023-11-24T06:18:52.422301+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6835/40191848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6833', 'nonce': 40191848}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 40191848}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T06:19:04.143 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T06:19:04.144 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-24T06:19:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.263+0000 7f70751c3700 1 Processor -- start 2023-11-24T06:19:04.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.263+0000 7f70751c3700 1 -- start start 2023-11-24T06:19:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f70751c3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f707015e080 0x7f70700b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f70751c3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70700af4d0 0x7f70700afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f70751c3700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f707006c000 0x7f707006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f70751c3700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70700af0b0 con 0x7f707015e080 2023-11-24T06:19:04.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f70751c3700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707006c9f0 con 0x7f707006c000 2023-11-24T06:19:04.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f70751c3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70700b05a0 con 0x7f70700af4d0 2023-11-24T06:19:04.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f707015e080 0x7f70700b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f707015e080 0x7f70700b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35680/0 (socket says 172.21.15.89:35680) 2023-11-24T06:19:04.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 -- 172.21.15.89:0/2112773269 learned_addr learned my addr 172.21.15.89:0/2112773269 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:04.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 -- 172.21.15.89:0/2112773269 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70700af4d0 msgr2=0x7f70700afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 --2- 172.21.15.89:0/2112773269 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70700af4d0 0x7f70700afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 -- 172.21.15.89:0/2112773269 >> v2:172.21.15.123:3300/0 conn(0x7f707006c000 msgr2=0x7f707006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706f59e700 1 --2- 172.21.15.89:0/2112773269 >> v2:172.21.15.123:3300/0 conn(0x7f707006c000 0x7f707006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 --2- 172.21.15.89:0/2112773269 >> v2:172.21.15.123:3300/0 conn(0x7f707006c000 0x7f707006c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 -- 172.21.15.89:0/2112773269 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70700ae430 con 0x7f707015e080 2023-11-24T06:19:04.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706ed9d700 1 --2- 172.21.15.89:0/2112773269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f707015e080 0x7f70700b5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f707009eea0 tx=0x7f7060009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ef7b32a3473be8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706dd9b700 1 -- 172.21.15.89:0/2112773269 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000b860 con 0x7f707015e080 2023-11-24T06:19:04.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706dd9b700 1 -- 172.21.15.89:0/2112773269 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706000b9c0 con 0x7f707015e080 2023-11-24T06:19:04.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.264+0000 7f706dd9b700 1 -- 172.21.15.89:0/2112773269 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000bd50 con 0x7f707015e080 2023-11-24T06:19:04.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 -- 172.21.15.89:0/2112773269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f707015e080 msgr2=0x7f70700b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 --2- 172.21.15.89:0/2112773269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f707015e080 0x7f70700b5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f707009eea0 tx=0x7f7060009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 -- 172.21.15.89:0/2112773269 shutdown_connections 2023-11-24T06:19:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 --2- 172.21.15.89:0/2112773269 >> v2:172.21.15.123:3300/0 conn(0x7f707006c000 0x7f707006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 --2- 172.21.15.89:0/2112773269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f707015e080 0x7f70700b5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 --2- 172.21.15.89:0/2112773269 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70700af4d0 0x7f70700afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 -- 172.21.15.89:0/2112773269 >> 172.21.15.89:0/2112773269 conn(0x7f7070071b90 msgr2=0x7f7070071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 -- 172.21.15.89:0/2112773269 shutdown_connections 2023-11-24T06:19:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.265+0000 7f70751c3700 1 -- 172.21.15.89:0/2112773269 wait complete. 2023-11-24T06:19:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.266+0000 7f70751c3700 1 Processor -- start 2023-11-24T06:19:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.266+0000 7f70751c3700 1 -- start start 2023-11-24T06:19:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f70751c3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f707006c000 0x7f7070157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f70751c3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70700af4d0 0x7f707015a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f70751c3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f707015e080 0x7f707015a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f70751c3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7070157a90 con 0x7f70700af4d0 2023-11-24T06:19:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f70751c3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7070144fa0 con 0x7f707015e080 2023-11-24T06:19:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f70751c3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70700cee20 con 0x7f707006c000 2023-11-24T06:19:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70700af4d0 0x7f707015a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70700af4d0 0x7f707015a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35686/0 (socket says 172.21.15.89:35686) 2023-11-24T06:19:04.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706e59c700 1 -- 172.21.15.89:0/245171066 learned_addr learned my addr 172.21.15.89:0/245171066 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:04.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706ed9d700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f707006c000 0x7f7070157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706f59e700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f707015e080 0x7f707015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706e59c700 1 -- 172.21.15.89:0/245171066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f707006c000 msgr2=0x7f7070157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706e59c700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f707006c000 0x7f7070157550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706e59c700 1 -- 172.21.15.89:0/245171066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f707015e080 msgr2=0x7f707015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706e59c700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f707015e080 0x7f707015a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.267+0000 7f706e59c700 1 -- 172.21.15.89:0/245171066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7060009550 con 0x7f70700af4d0 2023-11-24T06:19:04.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.268+0000 7f706e59c700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70700af4d0 0x7f707015a2d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f706400b500 tx=0x7f706400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.268+0000 7f70577fe700 1 -- 172.21.15.89:0/245171066 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064011830 con 0x7f70700af4d0 2023-11-24T06:19:04.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.268+0000 7f70577fe700 1 -- 172.21.15.89:0/245171066 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70640119b0 con 0x7f70700af4d0 2023-11-24T06:19:04.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.268+0000 7f70751c3700 1 -- 172.21.15.89:0/245171066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7070145240 con 0x7f70700af4d0 2023-11-24T06:19:04.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.268+0000 7f70577fe700 1 -- 172.21.15.89:0/245171066 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064011d60 con 0x7f70700af4d0 2023-11-24T06:19:04.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.268+0000 7f70751c3700 1 -- 172.21.15.89:0/245171066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070145800 con 0x7f70700af4d0 2023-11-24T06:19:04.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.269+0000 7f70577fe700 1 -- 172.21.15.89:0/245171066 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f706401a400 con 0x7f70700af4d0 2023-11-24T06:19:04.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.269+0000 7f70577fe700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f705c06aed0 0x7f705c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.270+0000 7f70577fe700 1 -- 172.21.15.89:0/245171066 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7064089670 con 0x7f70700af4d0 2023-11-24T06:19:04.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.270+0000 7f706ed9d700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f705c06aed0 0x7f705c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.270+0000 7f706ed9d700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f705c06aed0 0x7f705c06d390 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7060009870 tx=0x7f7060008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.272+0000 7f70813bcb80 1 Processor -- start 2023-11-24T06:19:04.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.272+0000 7f70813bcb80 1 -- start start 2023-11-24T06:19:04.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f70813bcb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e357ffd770 con 0x55e35809fed0 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f70813bcb80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e357fe4750 con 0x55e35808a080 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f70813bcb80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e357fe2070 con 0x55e358087490 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580d0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35694/0 (socket says 172.21.15.89:35694) 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 -- 172.21.15.89:0/1218146771 learned_addr learned my addr 172.21.15.89:0/1218146771 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706f59e700 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706ed9d700 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 -- 172.21.15.89:0/1218146771 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 msgr2=0x55e3580b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580b3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 -- 172.21.15.89:0/1218146771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 msgr2=0x55e3580d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580d0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 -- 172.21.15.89:0/1218146771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e357e5a5e0 con 0x55e35809fed0 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f706e59c700 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580d0d80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f70640601b0 tx=0x7f7064011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcefd19b5598ff09 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f7054ff9700 1 -- 172.21.15.89:0/1218146771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706406d070 con 0x55e35809fed0 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f7054ff9700 1 -- 172.21.15.89:0/1218146771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706405c530 con 0x55e35809fed0 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.273+0000 7f7054ff9700 1 -- 172.21.15.89:0/1218146771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70640606e0 con 0x55e35809fed0 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 -- 172.21.15.89:0/1218146771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 msgr2=0x55e3580d0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580d0d80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f70640601b0 tx=0x7f7064011000 comp rx=0 tx=0).stop 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 -- 172.21.15.89:0/1218146771 shutdown_connections 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 --2- 172.21.15.89:0/1218146771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580d0d80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 -- 172.21.15.89:0/1218146771 >> 172.21.15.89:0/1218146771 conn(0x55e357fb8310 msgr2=0x55e3580cee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 -- 172.21.15.89:0/1218146771 shutdown_connections 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.274+0000 7f70813bcb80 1 -- 172.21.15.89:0/1218146771 wait complete. 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 1 -- start start 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 1 Processor -- start 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580ddd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580a3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e357fdd960 con 0x55e35809fed0 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e357f97510 con 0x55e35808a080 2023-11-24T06:19:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e357f971b0 con 0x55e358087490 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f70813bcb80 10 client.0 _ll_register_callbacks cb 0x55e357ec2a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f706f59e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580a3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f706f59e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580a3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35702/0 (socket says 172.21.15.89:35702) 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.275+0000 7f706f59e700 1 -- 172.21.15.89:0/3969972337 learned_addr learned my addr 172.21.15.89:0/3969972337 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706e59c700 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706ed9d700 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580ddd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706f59e700 1 -- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 msgr2=0x55e3580ddd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706f59e700 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580ddd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706f59e700 1 -- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 msgr2=0x55e3580b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706f59e700 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706f59e700 1 -- 172.21.15.89:0/3969972337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070145800 con 0x55e35809fed0 2023-11-24T06:19:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706ed9d700 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580ddd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f706f59e700 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580a3160 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f706800f880 tx=0x7f706800fc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9345463305fe4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f704dffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f704dffb700 1 -- 172.21.15.89:0/3969972337 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706800dc80 con 0x55e35809fed0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f704dffb700 1 -- 172.21.15.89:0/3969972337 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f706800dde0 con 0x55e35809fed0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 1 -- 172.21.15.89:0/3969972337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e35807ca40 con 0x55e35809fed0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f704dffb700 1 -- 172.21.15.89:0/3969972337 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7068008a80 con 0x55e35809fed0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 10 client.4653 fetch_fsmap learned FSMap version 8 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 1 -- 172.21.15.89:0/3969972337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55e357e06d90 con 0x55e35809fed0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f704dffb700 1 -- 172.21.15.89:0/3969972337 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e35807ca40 con 0x55e35809fed0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f704dffb700 1 -- 172.21.15.89:0/3969972337 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7068003a20 con 0x55e35809fed0 2023-11-24T06:19:04.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 10 client.4653 fetch_fsmap finished waiting for FSMap version 8 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 10 client.4653 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 20 client.4653 populate_metadata read hostname 'smithi089' 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] conn(0x55e3580b3c10 0x55e3580e18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 4 client.4653 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.276+0000 7f70813bcb80 1 -- 172.21.15.89:0/3969972337 --> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e358058840 con 0x55e3580b3c10 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.277+0000 7f706ed9d700 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] conn(0x55e3580b3c10 0x55e3580e18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.277+0000 7f704dffb700 10 client.4653 ms_handle_connect on v2:172.21.15.123:6833/40191848 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.277+0000 7f706ed9d700 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] conn(0x55e3580b3c10 0x55e3580e18e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c363dab893d46b9 server_cookie=f5c3ab222b25edc in_seq=0 out_seq=0 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f704dffb700 1 -- 172.21.15.89:0/3969972337 <== mds.0 v2:172.21.15.123:6833/40191848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e358058840 con 0x55e3580b3c10 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f704dffb700 10 client.4653 handle_command_reply: tid=0 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 client.4653 shutdown 2023-11-24T06:19:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 -- 172.21.15.89:0/3969972337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 msgr2=0x55e3580a3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580a3160 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f706800f880 tx=0x7f706800fc40 comp rx=0 tx=0).stop 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 -- 172.21.15.89:0/3969972337 shutdown_connections 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] conn(0x55e3580b3c10 0x55e3580e18e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e35809fed0 0x55e3580a3160 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e358087490 0x55e3580ddd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 --2- 172.21.15.89:0/3969972337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f704dffb700 0 client.4653 ms_handle_reset on v2:172.21.15.123:6833/40191848 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 -- 172.21.15.89:0/3969972337 >> 172.21.15.89:0/3969972337 conn(0x55e357fb8310 msgr2=0x55e3580d2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 -- 172.21.15.89:0/3969972337 shutdown_connections 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 1 -- 172.21.15.89:0/3969972337 wait complete. 2023-11-24T06:19:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.278+0000 7f70813bcb80 20 client.4653 trim_cache size 0 max 16384 2023-11-24T06:19:04.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.296+0000 7f70813bcb80 1 Processor -- start 2023-11-24T06:19:04.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.296+0000 7f70813bcb80 1 -- start start 2023-11-24T06:19:04.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e3580b6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e35808a080 0x55e3580b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35809fed0 0x55e3580cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70813bcb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e357e5c8d0 con 0x55e358087490 2023-11-24T06:19:04.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70813bcb80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3580d1ca0 con 0x55e35809fed0 2023-11-24T06:19:04.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70813bcb80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3580b88d0 con 0x55e35808a080 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e3580b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e3580b6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35706/0 (socket says 172.21.15.89:35706) 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 -- 172.21.15.89:0/1057840766 learned_addr learned my addr 172.21.15.89:0/1057840766 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 -- 172.21.15.89:0/1057840766 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e35808a080 msgr2=0x55e3580b69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 --2- 172.21.15.89:0/1057840766 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e35808a080 0x55e3580b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 -- 172.21.15.89:0/1057840766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35809fed0 msgr2=0x55e3580cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 --2- 172.21.15.89:0/1057840766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35809fed0 0x55e3580cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 -- 172.21.15.89:0/1057840766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e357defc10 con 0x55e358087490 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f706ed9d700 1 --2- 172.21.15.89:0/1057840766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e3580b6470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f706000b0e0 tx=0x7f7060010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49e4f5778602060 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70557fa700 1 -- 172.21.15.89:0/1057840766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706002a040 con 0x55e358087490 2023-11-24T06:19:04.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70557fa700 1 -- 172.21.15.89:0/1057840766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706001f070 con 0x55e358087490 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.297+0000 7f70557fa700 1 -- 172.21.15.89:0/1057840766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70600325c0 con 0x55e358087490 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 -- 172.21.15.89:0/1057840766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 msgr2=0x55e3580b6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 --2- 172.21.15.89:0/1057840766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e3580b6470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f706000b0e0 tx=0x7f7060010000 comp rx=0 tx=0).stop 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 -- 172.21.15.89:0/1057840766 shutdown_connections 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 --2- 172.21.15.89:0/1057840766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35809fed0 0x55e3580cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 --2- 172.21.15.89:0/1057840766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e3580b6470 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 --2- 172.21.15.89:0/1057840766 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e35808a080 0x55e3580b69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 -- 172.21.15.89:0/1057840766 >> 172.21.15.89:0/1057840766 conn(0x55e357fb8310 msgr2=0x55e3580bfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 -- 172.21.15.89:0/1057840766 shutdown_connections 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 -- 172.21.15.89:0/1057840766 wait complete. 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 -- start start 2023-11-24T06:19:04.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.298+0000 7f70813bcb80 1 Processor -- start 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e358137900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f70813bcb80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e35809fed0 0x55e3580cd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f70813bcb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e358138340 con 0x55e358087490 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f70813bcb80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3580b4360 con 0x55e35808a080 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f70813bcb80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3580e0740 con 0x55e35809fed0 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f70813bcb80 10 client.0 _ll_register_callbacks cb 0x55e357ec2a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f706ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e358137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f706ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e358137900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35714/0 (socket says 172.21.15.89:35714) 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f706ed9d700 1 -- 172.21.15.89:0/2675756560 learned_addr learned my addr 172.21.15.89:0/2675756560 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f706ed9d700 1 -- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e35809fed0 msgr2=0x55e3580cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f706ed9d700 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e35809fed0 0x55e3580cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f706ed9d700 1 -- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 msgr2=0x55e3580ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f706ed9d700 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.299+0000 7f706ed9d700 1 -- 172.21.15.89:0/2675756560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70700ae430 con 0x55e358087490 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f706ed9d700 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e358137900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f706000bc80 tx=0x7f70600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fbc9739edfb2a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f704dffb700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f704dffb700 1 -- 172.21.15.89:0/2675756560 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706002a040 con 0x55e358087490 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f704dffb700 1 -- 172.21.15.89:0/2675756560 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70600327d0 con 0x55e358087490 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 1 -- 172.21.15.89:0/2675756560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f706405c530 con 0x55e358087490 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f704dffb700 1 -- 172.21.15.89:0/2675756560 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70600036a0 con 0x55e358087490 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 10 client.4659 fetch_fsmap learned FSMap version 8 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 1 -- 172.21.15.89:0/2675756560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55e357e2d200 con 0x55e358087490 2023-11-24T06:19:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f704dffb700 1 -- 172.21.15.89:0/2675756560 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706405c530 con 0x55e358087490 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f704dffb700 1 -- 172.21.15.89:0/2675756560 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7060003800 con 0x55e358087490 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 10 client.4659 fetch_fsmap finished waiting for FSMap version 8 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 10 client.4659 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 20 client.4659 populate_metadata read hostname 'smithi089' 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] conn(0x55e35813b960 0x55e35813bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 4 client.4659 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f70813bcb80 1 -- 172.21.15.89:0/2675756560 --> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e357fa3f80 con 0x55e35813b960 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.300+0000 7f706e59c700 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] conn(0x55e35813b960 0x55e35813bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f704dffb700 10 client.4659 ms_handle_connect on v2:172.21.15.123:6833/40191848 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f706e59c700 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] conn(0x55e35813b960 0x55e35813bd40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8156af1bbeac00a9 server_cookie=6dd265eae8fa6fd3 in_seq=0 out_seq=0 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f704dffb700 1 -- 172.21.15.89:0/2675756560 <== mds.0 v2:172.21.15.123:6833/40191848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55e357fa3f80 con 0x55e35813b960 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f704dffb700 10 client.4659 handle_command_reply: tid=0 2023-11-24T06:19:04.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 client.4659 shutdown 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 -- 172.21.15.89:0/2675756560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 msgr2=0x55e358137900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e358137900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f706000bc80 tx=0x7f70600047c0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 -- 172.21.15.89:0/2675756560 shutdown_connections 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:6833/40191848,v1:172.21.15.123:6835/40191848] conn(0x55e35813b960 0x55e35813bd40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55e35809fed0 0x55e3580cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e358087490 0x55e358137900 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f704dffb700 0 client.4659 ms_handle_reset on v2:172.21.15.123:6833/40191848 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 --2- 172.21.15.89:0/2675756560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55e35808a080 0x55e3580ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 -- 172.21.15.89:0/2675756560 >> 172.21.15.89:0/2675756560 conn(0x55e357fb8310 msgr2=0x55e3580d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 -- 172.21.15.89:0/2675756560 shutdown_connections 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 1 -- 172.21.15.89:0/2675756560 wait complete. 2023-11-24T06:19:04.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.301+0000 7f70813bcb80 20 client.4659 trim_cache size 0 max 16384 2023-11-24T06:19:04.317 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:19:04.317 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "be2488f4-68a7-42cc-93d1-ae9c6921fb48", 2023-11-24T06:19:04.317 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2023-11-24T06:19:04.317 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 4290, 2023-11-24T06:19:04.317 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-11-24T06:19:04.317 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-11-24T06:19:04.317 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-11-24T06:19:04.317 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 14.285230737999999, 2023-11-24T06:19:04.318 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 8, 2023-11-24T06:19:04.318 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 21, 2023-11-24T06:19:04.318 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 21, 2023-11-24T06:19:04.318 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 21.979513604000001 2023-11-24T06:19:04.318 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:19:04.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 -- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f705c06aed0 msgr2=0x7f705c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f705c06aed0 0x7f705c06d390 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7060009870 tx=0x7f7060008030 comp rx=0 tx=0).stop 2023-11-24T06:19:04.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 -- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70700af4d0 msgr2=0x7f707015a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70700af4d0 0x7f707015a2d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f706400b500 tx=0x7f706400b8c0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 -- 172.21.15.89:0/245171066 shutdown_connections 2023-11-24T06:19:04.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f705c06aed0 0x7f705c06d390 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f707015e080 0x7f707015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f707006c000 0x7f7070157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 --2- 172.21.15.89:0/245171066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70700af4d0 0x7f707015a2d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 -- 172.21.15.89:0/245171066 >> 172.21.15.89:0/245171066 conn(0x7f7070071b90 msgr2=0x7f707015d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:04.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 -- 172.21.15.89:0/245171066 shutdown_connections 2023-11-24T06:19:04.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.302+0000 7f70751c3700 1 -- 172.21.15.89:0/245171066 wait complete. 2023-11-24T06:19:04.320 INFO:teuthology.run_tasks:Running task kclient... 2023-11-24T06:19:04.345 INFO:tasks.kclient:Mounting kernel clients... 2023-11-24T06:19:04.345 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-11-24T06:19:04.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:04.346 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:19:04.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:04.363 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-11-24T06:19:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2023-11-24T06:19:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:19:04.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:04.433 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:19:04.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:04.492 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-11-24T06:19:04.558 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2023-11-24T06:19:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:19:04.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:04.560 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:19:04.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:04.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2023-11-24T06:19:04.657 INFO:teuthology.orchestra.run.smithi123.stderr:Cannot find device "ceph-brx" 2023-11-24T06:19:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:19:04.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:04.660 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:19:04.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:04.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2023-11-24T06:19:04.793 INFO:teuthology.orchestra.run.smithi123.stderr:Cannot find device "ceph-brx" 2023-11-24T06:19:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:19:04.795 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-11-24T06:19:04.795 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T06:19:04.795 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:04.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:19:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.912+0000 7fc176eec700 1 Processor -- start 2023-11-24T06:19:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.912+0000 7fc176eec700 1 -- start start 2023-11-24T06:19:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc176eec700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc170147780 0x7fc17015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc176eec700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc17006dc20 0x7fc170073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc176eec700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17015eb00 con 0x7fc1701473a0 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc176eec700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17015ede0 con 0x7fc17006dc20 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc176eec700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17015f0c0 con 0x7fc170147780 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc175489700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc17006dc20 0x7fc170073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc174c88700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc1701473a0 0x7fc17006c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39224/0 (socket says 172.21.15.89:39224) 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc174c88700 1 -- 172.21.15.89:0/439681567 learned_addr learned my addr 172.21.15.89:0/439681567 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc175489700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc17006dc20 0x7fc170073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35722/0 (socket says 172.21.15.89:35722) 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc175489700 1 -- 172.21.15.89:0/439681567 >> v2:172.21.15.123:3300/0 conn(0x7fc170147780 msgr2=0x7fc17015e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc16ffff700 1 --2- 172.21.15.89:0/439681567 >> v2:172.21.15.123:3300/0 conn(0x7fc170147780 0x7fc17015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc16f7fe700 1 -- 172.21.15.89:0/439681567 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516347183 0 0) 0x7fc17015eb00 con 0x7fc1701473a0 2023-11-24T06:19:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc175489700 1 --2- 172.21.15.89:0/439681567 >> v2:172.21.15.123:3300/0 conn(0x7fc170147780 0x7fc17015e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc175489700 1 -- 172.21.15.89:0/439681567 >> v1:172.21.15.89:6789/0 conn(0x7fc1701473a0 legacy=0x7fc17006c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc175489700 1 -- 172.21.15.89:0/439681567 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17015f3a0 con 0x7fc17006dc20 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.913+0000 7fc175489700 1 --2- 172.21.15.89:0/439681567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17006dc20 0x7fc170073f90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc170147d30 tx=0x7fc16800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1819844807f763 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc16f7fe700 1 -- 172.21.15.89:0/439681567 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16800e070 con 0x7fc17006dc20 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc16f7fe700 1 -- 172.21.15.89:0/439681567 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16800bbd0 con 0x7fc17006dc20 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc16f7fe700 1 -- 172.21.15.89:0/439681567 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc168012430 con 0x7fc17006dc20 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc176eec700 1 -- 172.21.15.89:0/439681567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17006dc20 msgr2=0x7fc170073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc176eec700 1 --2- 172.21.15.89:0/439681567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17006dc20 0x7fc170073f90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc170147d30 tx=0x7fc16800b330 comp rx=0 tx=0).stop 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc176eec700 1 -- 172.21.15.89:0/439681567 shutdown_connections 2023-11-24T06:19:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc176eec700 1 --2- 172.21.15.89:0/439681567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17006dc20 0x7fc170073f90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc176eec700 1 --2- 172.21.15.89:0/439681567 >> v2:172.21.15.123:3300/0 conn(0x7fc170147780 0x7fc17015e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc176eec700 1 -- 172.21.15.89:0/439681567 >> 172.21.15.89:0/439681567 conn(0x7fc1700b3130 msgr2=0x7fc1700d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc176eec700 1 -- 172.21.15.89:0/439681567 shutdown_connections 2023-11-24T06:19:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.914+0000 7fc176eec700 1 -- 172.21.15.89:0/439681567 wait complete. 2023-11-24T06:19:04.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.915+0000 7fc176eec700 1 Processor -- start 2023-11-24T06:19:04.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.915+0000 7fc176eec700 1 -- start start 2023-11-24T06:19:04.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc176eec700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc17006dc20 0x7fc17013f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc176eec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1701473a0 0x7fc170153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc176eec700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc170147780 0x7fc170141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc176eec700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1701406d0 con 0x7fc1701473a0 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc176eec700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17013f860 con 0x7fc17006dc20 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc176eec700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17013fac0 con 0x7fc170147780 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc16ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1701473a0 0x7fc170153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc16ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1701473a0 0x7fc170153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35726/0 (socket says 172.21.15.89:35726) 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.916+0000 7fc16ffff700 1 -- 172.21.15.89:0/3355701234 learned_addr learned my addr 172.21.15.89:0/3355701234 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc174c88700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc17006dc20 0x7fc17013f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc175489700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc170147780 0x7fc170141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16ffff700 1 -- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc170147780 msgr2=0x7fc170141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16ffff700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc170147780 0x7fc170141160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16ffff700 1 -- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc17006dc20 msgr2=0x7fc17013f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16ffff700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc17006dc20 0x7fc17013f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16ffff700 1 -- 172.21.15.89:0/3355701234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc168009a70 con 0x7fc1701473a0 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc174c88700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc17006dc20 0x7fc17013f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc175489700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc170147780 0x7fc170141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16ffff700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1701473a0 0x7fc170153960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c7f0 tx=0x7fc15c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16d7fa700 1 -- 172.21.15.89:0/3355701234 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c004220 con 0x7fc1701473a0 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16d7fa700 1 -- 172.21.15.89:0/3355701234 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c004380 con 0x7fc1701473a0 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17013fda0 con 0x7fc1701473a0 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc16d7fa700 1 -- 172.21.15.89:0/3355701234 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0036a0 con 0x7fc1701473a0 2023-11-24T06:19:04.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.917+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc170140360 con 0x7fc1701473a0 2023-11-24T06:19:04.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.918+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc170140880 con 0x7fc1701473a0 2023-11-24T06:19:04.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.918+0000 7fc16d7fa700 1 -- 172.21.15.89:0/3355701234 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc15c003800 con 0x7fc1701473a0 2023-11-24T06:19:04.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.919+0000 7fc16d7fa700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc15406aed0 0x7fc15406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:04.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.919+0000 7fc16d7fa700 1 -- 172.21.15.89:0/3355701234 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc15c089040 con 0x7fc1701473a0 2023-11-24T06:19:04.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.919+0000 7fc174c88700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc15406aed0 0x7fc15406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:04.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.919+0000 7fc174c88700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc15406aed0 0x7fc15406d390 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc164000f60 tx=0x7fc164009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:04.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:04.923+0000 7fc16d7fa700 1 -- 172.21.15.89:0/3355701234 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc170140880 con 0x7fc1701473a0 2023-11-24T06:19:05.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.158+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc170067f60 con 0x7fc1701473a0 2023-11-24T06:19:05.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.158+0000 7fc16d7fa700 1 -- 172.21.15.89:0/3355701234 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc15c01e060 con 0x7fc1701473a0 2023-11-24T06:19:05.161 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:19:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc15406aed0 msgr2=0x7fc15406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc15406aed0 0x7fc15406d390 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc164000f60 tx=0x7fc164009040 comp rx=0 tx=0).stop 2023-11-24T06:19:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1701473a0 msgr2=0x7fc170153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1701473a0 0x7fc170153960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c7f0 tx=0x7fc15c00cbb0 comp rx=0 tx=0).stop 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 shutdown_connections 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc170147780 0x7fc170141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc15406aed0 0x7fc15406d390 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc17006dc20 0x7fc17013f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 --2- 172.21.15.89:0/3355701234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1701473a0 0x7fc170153960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 >> 172.21.15.89:0/3355701234 conn(0x7fc1700b3130 msgr2=0x7fc17006b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 shutdown_connections 2023-11-24T06:19:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:05.160+0000 7fc176eec700 1 -- 172.21.15.89:0/3355701234 wait complete. 2023-11-24T06:19:05.175 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T06:19:05.175 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:19:05.176 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2023-11-24T06:19:05.176 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T06:19:05.176 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:05.176 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T06:19:05.176 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T06:19:05.176 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:19:05.177 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:19:05.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.177 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2023-11-24T06:19:05.193 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:05.195 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:19:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:19:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42415sec preferred_lft 42415sec 2023-11-24T06:19:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4376:7845:d8a4:7bc1/64 scope link noprefixroute 2023-11-24T06:19:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:05.195 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-24T06:19:05.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.196 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:05.196 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:05.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2023-11-24T06:19:05.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2023-11-24T06:19:05.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2023-11-24T06:19:05.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-24T06:19:05.196 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:05.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.364 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:19:05.391 INFO:teuthology.orchestra.run.smithi089.stdout:1 2023-11-24T06:19:05.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.393 DEBUG:teuthology.orchestra.run.smithi089:> route 2023-11-24T06:19:05.463 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2023-11-24T06:19:05.463 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:19:05.464 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:19:05.464 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:19:05.464 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:19:05.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.465 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:05.465 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:05.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:19:05.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:19:05.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:19:05.466 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:05.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.688 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:19:05.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.705 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2023-11-24T06:19:05.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.762 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:05.762 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:05.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:19:05.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-24T06:19:05.762 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:05.855 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-24T06:19:05.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:05.856 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:05.856 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:05.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-24T06:19:05.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-24T06:19:05.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-24T06:19:05.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-24T06:19:05.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-24T06:19:05.856 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:06.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:06.064 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:06.064 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:06.064 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2023-11-24T06:19:06.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2023-11-24T06:19:06.065 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:06.178 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:06.178 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:06.194 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:19:06.195 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:06.195 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:06.251 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:19:06.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:06.251 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:19:06.343 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:19:06.343 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:19:06.343 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:06.453 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:19:06.454 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:19:06.454 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:06.454 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:06.454 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:06.484 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:19:06.485 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:19:06.485 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:06.485 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:06.485 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:06.485 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:19:06.485 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:19:06.485 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:19:06.488 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:19:06.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:06.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:06.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:06.522 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:06.614 INFO:teuthology.orchestra.run.smithi089.stdout:client4671 2023-11-24T06:19:06.615 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:06.615 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:06.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:06.632 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:06.720 INFO:teuthology.orchestra.run.smithi089.stdout:client4671 2023-11-24T06:19:06.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:06.720 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:06.767 INFO:teuthology.orchestra.run.smithi089.stdout:client4671 2023-11-24T06:19:06.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:06.768 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/be2488f4-68a7-42cc-93d1-ae9c6921fb48.client4671/status') 2023-11-24T06:19:06.821 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.4671 (3)192.168.0.1:0/4213059101 2023-11-24T06:19:06.821 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2023-11-24T06:19:06.821 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2023-11-24T06:19:06.822 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2023-11-24T06:19:06.822 INFO:teuthology.orchestra.run.smithi089.stderr:69 bytes copied, 2.2059e-05 s, 3.1 MB/s 2023-11-24T06:19:06.823 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2023-11-24T06:19:06.823 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T06:19:06.823 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-24T06:19:06.824 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:19:06.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f522e60e700 1 Processor -- start 2023-11-24T06:19:06.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f522e60e700 1 -- start start 2023-11-24T06:19:06.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f522e60e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f52280b3370 0x7f52281604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f522e60e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280d1c70 0x7f52280d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f522e60e700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52280b22d0 con 0x7f522815bdd0 2023-11-24T06:19:06.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f522e60e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52280b2f50 con 0x7f52280b3370 2023-11-24T06:19:06.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f522e60e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52280d2640 con 0x7f52280d1c70 2023-11-24T06:19:06.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f5227fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f522815bdd0 0x7f52280b1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39244/0 (socket says 172.21.15.89:39244) 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.941+0000 7f5227fff700 1 -- 172.21.15.89:0/3421492455 learned_addr learned my addr 172.21.15.89:0/3421492455 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354323730 0 0) 0x7f52280b22d0 con 0x7f522815bdd0 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f52277fe700 1 --2- 172.21.15.89:0/3421492455 >> v2:172.21.15.123:3300/0 conn(0x7f52280b3370 0x7f52281604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52140035b0 con 0x7f522815bdd0 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f522cbab700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280d1c70 0x7f52280d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234662188 0 0) 0x7f52140035b0 con 0x7f522815bdd0 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280d1c70 msgr2=0x7f52280d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280d1c70 0x7f52280d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 >> v2:172.21.15.123:3300/0 conn(0x7f52280b3370 msgr2=0x7f52281604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 --2- 172.21.15.89:0/3421492455 >> v2:172.21.15.123:3300/0 conn(0x7f52280b3370 0x7f52281604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52280b25b0 con 0x7f522815bdd0 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f5218001670 con 0x7f522815bdd0 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 >> v1:172.21.15.89:6789/0 conn(0x7f522815bdd0 legacy=0x7f52280b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280d1c70 0x7f5214003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f5214004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5214004d90 0x7f5214005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52280b2f50 con 0x7f5214003f80 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52280d2640 con 0x7f5214004d90 2023-11-24T06:19:06.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521400ba10 con 0x7f52280d1c70 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f52277fe700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f5214004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f5227fff700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280d1c70 0x7f5214003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f522cbab700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5214004d90 0x7f5214005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f52277fe700 1 -- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280d1c70 msgr2=0x7f5214003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f52277fe700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280d1c70 0x7f5214003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f52277fe700 1 -- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5214004d90 msgr2=0x7f5214005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f52277fe700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5214004d90 0x7f5214005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.942+0000 7f52277fe700 1 -- 172.21.15.89:0/3421492455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52140035b0 con 0x7f5214003f80 2023-11-24T06:19:06.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f52277fe700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f5214004850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5210009b60 tx=0x7f5214005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f7d54eedcb7b5b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521000e660 con 0x7f5214003f80 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521000e7c0 con 0x7f5214003f80 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f5226ffd700 1 -- 172.21.15.89:0/3421492455 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521000eb30 con 0x7f5214003f80 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f522e60e700 1 -- 172.21.15.89:0/3421492455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 msgr2=0x7f5214004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f522e60e700 1 --2- 172.21.15.89:0/3421492455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f5214004850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5210009b60 tx=0x7f5214005880 comp rx=0 tx=0).stop 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f522cbab700 1 -- 172.21.15.89:0/3421492455 reap_dead start 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f522e60e700 1 -- 172.21.15.89:0/3421492455 shutdown_connections 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f522e60e700 1 -- 172.21.15.89:0/3421492455 >> 172.21.15.89:0/3421492455 conn(0x7f52280731a0 msgr2=0x7f522809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f522e60e700 1 -- 172.21.15.89:0/3421492455 shutdown_connections 2023-11-24T06:19:06.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.943+0000 7f522e60e700 1 -- 172.21.15.89:0/3421492455 wait complete. 2023-11-24T06:19:06.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.945+0000 7f522e60e700 1 Processor -- start 2023-11-24T06:19:06.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.945+0000 7f522e60e700 1 -- start start 2023-11-24T06:19:06.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522e60e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f52280b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522e60e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280b5400 0x7f52281569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522e60e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52280cd6d0 0x7f52280cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522e60e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52280ce090 con 0x7f5214003f80 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522e60e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52280b5910 con 0x7f52280cd6d0 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522e60e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52280b5bf0 con 0x7f52280b5400 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f52280b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f52280b48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35752/0 (socket says 172.21.15.89:35752) 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 -- 172.21.15.89:0/1200256464 learned_addr learned my addr 172.21.15.89:0/1200256464 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522cbab700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52280cd6d0 0x7f52280cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f52277fe700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280b5400 0x7f52281569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 -- 172.21.15.89:0/1200256464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280b5400 msgr2=0x7f52281569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:06.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280b5400 0x7f52281569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 -- 172.21.15.89:0/1200256464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52280cd6d0 msgr2=0x7f52280cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52280cd6d0 0x7f52280cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 -- 172.21.15.89:0/1200256464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52280b5ed0 con 0x7f5214003f80 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5227fff700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f52280b48d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5218007340 tx=0x7f5218007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5224ff9700 1 -- 172.21.15.89:0/1200256464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521800f6c0 con 0x7f5214003f80 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5224ff9700 1 -- 172.21.15.89:0/1200256464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521800f820 con 0x7f5214003f80 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52280c6c70 con 0x7f5214003f80 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f5224ff9700 1 -- 172.21.15.89:0/1200256464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521800fbd0 con 0x7f5214003f80 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.946+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52280b61b0 con 0x7f5214003f80 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.947+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52280c6fe0 con 0x7f5214003f80 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.948+0000 7f5224ff9700 1 -- 172.21.15.89:0/1200256464 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f521800fd30 con 0x7f5214003f80 2023-11-24T06:19:06.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.948+0000 7f5224ff9700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f520806aed0 0x7f520806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:06.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.948+0000 7f5224ff9700 1 -- 172.21.15.89:0/1200256464 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5218089590 con 0x7f5214003f80 2023-11-24T06:19:06.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.948+0000 7f52277fe700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f520806aed0 0x7f520806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:06.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.948+0000 7f52277fe700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f520806aed0 0x7f520806d390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f52100090b0 tx=0x7f5210008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:06.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:06.952+0000 7f5224ff9700 1 -- 172.21.15.89:0/1200256464 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5218054640 con 0x7f5214003f80 2023-11-24T06:19:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.189+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f52280d01b0 con 0x7f5214003f80 2023-11-24T06:19:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.189+0000 7f5224ff9700 1 -- 172.21.15.89:0/1200256464 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5218057c60 con 0x7f5214003f80 2023-11-24T06:19:07.192 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f520806aed0 msgr2=0x7f520806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f520806aed0 0x7f520806d390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f52100090b0 tx=0x7f5210008030 comp rx=0 tx=0).stop 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 msgr2=0x7f52280b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f52280b48d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5218007340 tx=0x7f5218007650 comp rx=0 tx=0).stop 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 shutdown_connections 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f520806aed0 0x7f520806d390 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52280cd6d0 0x7f52280cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5214003f80 0x7f52280b48d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 --2- 172.21.15.89:0/1200256464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f52280b5400 0x7f52281569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 >> 172.21.15.89:0/1200256464 conn(0x7f52280731a0 msgr2=0x7f52280aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.191+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 shutdown_connections 2023-11-24T06:19:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:07.192+0000 7f522e60e700 1 -- 172.21.15.89:0/1200256464 wait complete. 2023-11-24T06:19:07.206 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T06:19:07.206 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:19:07.206 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2023-11-24T06:19:07.207 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-11-24T06:19:07.207 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-24T06:19:07.207 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T06:19:07.207 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T06:19:07.207 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:19:07.207 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-11-24T06:19:07.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.207 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2023-11-24T06:19:07.224 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:19:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42413sec preferred_lft 42413sec 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4376:7845:d8a4:7bc1/64 scope link noprefixroute 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 12:de:17:7b:11:37 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::3cfa:7aff:fed2:90c8/64 scope link 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.226 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:19:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 12:de:17:7b:11:37 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:19:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::10de:17ff:fe7b:1137/64 scope link tentative 2023-11-24T06:19:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.227 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:19:07.289 INFO:teuthology.orchestra.run.smithi089.stdout:1 2023-11-24T06:19:07.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.291 DEBUG:teuthology.orchestra.run.smithi089:> route 2023-11-24T06:19:07.348 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2023-11-24T06:19:07.348 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:19:07.349 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:19:07.349 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:19:07.349 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:19:07.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.349 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:07.349 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:07.349 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:19:07.350 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:19:07.350 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:19:07.350 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:07.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.452 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:19:07.468 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T06:19:07.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.469 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2023-11-24T06:19:07.525 INFO:teuthology.orchestra.run.smithi089.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-11-24T06:19:07.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.525 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:07.525 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:07.526 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T06:19:07.526 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-11-24T06:19:07.526 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:07.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 7e:9c:c1:04:be:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T06:19:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T06:19:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::7c9c:c1ff:fe04:bead/64 scope link 2023-11-24T06:19:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 7e:9c:c1:04:be:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T06:19:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T06:19:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::7c9c:c1ff:fe04:bead/64 scope link 2023-11-24T06:19:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:07.754 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-11-24T06:19:07.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:07.754 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:07.754 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:07.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-11-24T06:19:07.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-24T06:19:07.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-11-24T06:19:07.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-11-24T06:19:07.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-11-24T06:19:07.756 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:08.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:08.003 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:08.003 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:08.003 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.1 up 2023-11-24T06:19:08.004 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.1 master ceph-brx 2023-11-24T06:19:08.004 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:08.132 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:08.132 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-11-24T06:19:08.153 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-11-24T06:19:08.154 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:08.154 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-11-24T06:19:08.209 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:19:08.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:08.210 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-24T06:19:08.294 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-24T06:19:08.295 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-24T06:19:08.295 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:08.304 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-24T06:19:08.304 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-24T06:19:08.304 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:08.304 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:08.304 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:08.379 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-24T06:19:08.379 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-24T06:19:08.379 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:08.379 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:08.379 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:08.379 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:19:08.379 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:19:08.380 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:19:08.381 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:19:08.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:08.382 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-11-24T06:19:08.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:08.408 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:08.480 INFO:teuthology.orchestra.run.smithi089.stdout:client4451 2023-11-24T06:19:08.481 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:08.481 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:08.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:08.538 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:08.610 INFO:teuthology.orchestra.run.smithi089.stdout:client4451 2023-11-24T06:19:08.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:08.610 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:08.681 INFO:teuthology.orchestra.run.smithi089.stdout:client4451 2023-11-24T06:19:08.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:08.682 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/be2488f4-68a7-42cc-93d1-ae9c6921fb48.client4451/status') 2023-11-24T06:19:08.763 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.4451 (3)172.21.15.89:0/2288876377 2023-11-24T06:19:08.763 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2023-11-24T06:19:08.764 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2023-11-24T06:19:08.764 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2023-11-24T06:19:08.764 INFO:teuthology.orchestra.run.smithi089.stderr:70 bytes copied, 2.2316e-05 s, 3.1 MB/s 2023-11-24T06:19:08.764 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-11-24T06:19:08.764 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T06:19:08.764 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:08.765 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:19:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.921+0000 7fb832a58700 1 Processor -- start 2023-11-24T06:19:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.921+0000 7fb832a58700 1 -- start start 2023-11-24T06:19:08.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb832a58700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb82c0d1570 0x7fb82c0d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:08.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb832a58700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0d2d10 con 0x7fb82c0b2fc0 2023-11-24T06:19:08.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb832a58700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0cefd0 con 0x7fb82c15dfc0 2023-11-24T06:19:08.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb832a58700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c0cf2b0 con 0x7fb82c0d1570 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb82b7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb82c0d1570 0x7fb82c0d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb830ff5700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb82c0b2fc0 0x7fb82c0d3a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39268/0 (socket says 172.21.15.89:39268) 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb830ff5700 1 -- 172.21.15.89:0/1913732657 learned_addr learned my addr 172.21.15.89:0/1913732657 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb82b7fe700 1 -- 172.21.15.89:0/1913732657 >> v1:172.21.15.123:6789/0 conn(0x7fb82c15dfc0 legacy=0x7fb82c0d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb82b7fe700 1 -- 172.21.15.89:0/1913732657 >> v1:172.21.15.89:6789/0 conn(0x7fb82c0b2fc0 legacy=0x7fb82c0d3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb82b7fe700 1 -- 172.21.15.89:0/1913732657 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c0d2fb0 con 0x7fb82c0d1570 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.922+0000 7fb82b7fe700 1 --2- 172.21.15.89:0/1913732657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c0d1570 0x7fb82c0d34d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009840 tx=0x7fb81c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9906f4a0cb33820d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb82affd700 1 -- 172.21.15.89:0/1913732657 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c011830 con 0x7fb82c0d1570 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb82affd700 1 -- 172.21.15.89:0/1913732657 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c011990 con 0x7fb82c0d1570 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb82affd700 1 -- 172.21.15.89:0/1913732657 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c011d20 con 0x7fb82c0d1570 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb832a58700 1 -- 172.21.15.89:0/1913732657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c0d1570 msgr2=0x7fb82c0d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:08.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb832a58700 1 --2- 172.21.15.89:0/1913732657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c0d1570 0x7fb82c0d34d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009840 tx=0x7fb81c009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb832a58700 1 -- 172.21.15.89:0/1913732657 shutdown_connections 2023-11-24T06:19:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb832a58700 1 --2- 172.21.15.89:0/1913732657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c0d1570 0x7fb82c0d34d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb832a58700 1 -- 172.21.15.89:0/1913732657 >> 172.21.15.89:0/1913732657 conn(0x7fb82c06b410 msgr2=0x7fb82c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb832a58700 1 -- 172.21.15.89:0/1913732657 shutdown_connections 2023-11-24T06:19:08.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.923+0000 7fb832a58700 1 -- 172.21.15.89:0/1913732657 wait complete. 2023-11-24T06:19:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.925+0000 7fb832a58700 1 Processor -- start 2023-11-24T06:19:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.925+0000 7fb832a58700 1 -- start start 2023-11-24T06:19:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb832a58700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb82c0b2fc0 0x7fb82c1e9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb832a58700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb82c0d1570 0x7fb82c0b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb832a58700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c15dfc0 0x7fb82c1e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb832a58700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c1e9850 con 0x7fb82c15dfc0 2023-11-24T06:19:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb832a58700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0c8a50 con 0x7fb82c0d1570 2023-11-24T06:19:08.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb832a58700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c1e8bd0 con 0x7fb82c0b2fc0 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c15dfc0 0x7fb82c1e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c15dfc0 0x7fb82c1e6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35774/0 (socket says 172.21.15.89:35774) 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 -- 172.21.15.89:0/3384398727 learned_addr learned my addr 172.21.15.89:0/3384398727 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb82bfff700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb82c0b2fc0 0x7fb82c1e9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb82b7fe700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb82c0d1570 0x7fb82c0b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 -- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb82c0b2fc0 msgr2=0x7fb82c1e9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb82c0b2fc0 0x7fb82c1e9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 -- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb82c0d1570 msgr2=0x7fb82c0b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb82c0d1570 0x7fb82c0b8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 -- 172.21.15.89:0/3384398727 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c0094a0 con 0x7fb82c15dfc0 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb82bfff700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb82c0b2fc0 0x7fb82c1e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:08.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb830ff5700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c15dfc0 0x7fb82c1e6bd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb82400ca80 tx=0x7fb82400ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb828ff9700 1 -- 172.21.15.89:0/3384398727 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82400aaa0 con 0x7fb82c15dfc0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.926+0000 7fb828ff9700 1 -- 172.21.15.89:0/3384398727 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82400ac00 con 0x7fb82c15dfc0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.927+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c1e8eb0 con 0x7fb82c15dfc0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.927+0000 7fb828ff9700 1 -- 172.21.15.89:0/3384398727 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824012400 con 0x7fb82c15dfc0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.927+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c1e9470 con 0x7fb82c15dfc0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.927+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82c1e9ac0 con 0x7fb82c15dfc0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.928+0000 7fb828ff9700 1 -- 172.21.15.89:0/3384398727 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb82401b3b0 con 0x7fb82c15dfc0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.928+0000 7fb828ff9700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb81806aed0 0x7fb81806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.928+0000 7fb82bfff700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb81806aed0 0x7fb81806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:08.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.928+0000 7fb828ff9700 1 -- 172.21.15.89:0/3384398727 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8240890b0 con 0x7fb82c15dfc0 2023-11-24T06:19:08.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.929+0000 7fb82bfff700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb81806aed0 0x7fb81806d390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0cdc00 tx=0x7fb820009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:08.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:08.933+0000 7fb828ff9700 1 -- 172.21.15.89:0/3384398727 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8240542a0 con 0x7fb82c15dfc0 2023-11-24T06:19:09.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.166+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb82c067f60 con 0x7fb82c15dfc0 2023-11-24T06:19:09.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.166+0000 7fb828ff9700 1 -- 172.21.15.89:0/3384398727 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb8240578c0 con 0x7fb82c15dfc0 2023-11-24T06:19:09.169 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:19:09.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb81806aed0 msgr2=0x7fb81806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:09.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb81806aed0 0x7fb81806d390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0cdc00 tx=0x7fb820009040 comp rx=0 tx=0).stop 2023-11-24T06:19:09.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c15dfc0 msgr2=0x7fb82c1e6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:09.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c15dfc0 0x7fb82c1e6bd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb82400ca80 tx=0x7fb82400ce40 comp rx=0 tx=0).stop 2023-11-24T06:19:09.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 shutdown_connections 2023-11-24T06:19:09.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb82c15dfc0 0x7fb82c1e6bd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:09.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb82c0b2fc0 0x7fb82c1e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:09.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb81806aed0 0x7fb81806d390 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:09.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 --2- 172.21.15.89:0/3384398727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb82c0d1570 0x7fb82c0b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:09.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 >> 172.21.15.89:0/3384398727 conn(0x7fb82c06b410 msgr2=0x7fb82c0b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:09.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.168+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 shutdown_connections 2023-11-24T06:19:09.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:09.169+0000 7fb832a58700 1 -- 172.21.15.89:0/3384398727 wait complete. 2023-11-24T06:19:09.184 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T06:19:09.184 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:19:09.184 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2023-11-24T06:19:09.184 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T06:19:09.184 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:09.184 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T06:19:09.184 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T06:19:09.184 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:19:09.185 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:19:09.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.185 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2023-11-24T06:19:09.206 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:09.207 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:19:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:19:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42402sec preferred_lft 42402sec 2023-11-24T06:19:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::a981:dd7b:bd0d:851a/64 scope link noprefixroute 2023-11-24T06:19:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:09.208 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-24T06:19:09.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.209 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:09.209 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:09.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add name ceph-brx type bridge 2023-11-24T06:19:09.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr flush dev ceph-brx 2023-11-24T06:19:09.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set ceph-brx up 2023-11-24T06:19:09.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-24T06:19:09.209 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:09.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.381 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:19:09.410 INFO:teuthology.orchestra.run.smithi123.stdout:1 2023-11-24T06:19:09.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.411 DEBUG:teuthology.orchestra.run.smithi123:> route 2023-11-24T06:19:09.485 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2023-11-24T06:19:09.485 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:19:09.485 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:19:09.485 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:19:09.485 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:19:09.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.485 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:09.486 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:09.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:19:09.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:19:09.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:19:09.486 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:09.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.748 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:19:09.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.765 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2023-11-24T06:19:09.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.822 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:09.822 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:09.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:19:09.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-11-24T06:19:09.822 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:09.916 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-11-24T06:19:09.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:09.916 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:09.916 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:09.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-11-24T06:19:09.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-24T06:19:09.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-11-24T06:19:09.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-11-24T06:19:09.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-11-24T06:19:09.917 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:10.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:10.116 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:10.117 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:10.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2023-11-24T06:19:10.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.0 master ceph-brx 2023-11-24T06:19:10.117 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:10.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:10.217 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:10.236 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:19:10.237 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:10.237 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:10.292 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:19:10.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:10.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:19:10.406 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:19:10.406 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:19:10.407 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:10.502 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:19:10.502 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:19:10.502 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:10.502 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:10.502 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:10.532 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:19:10.532 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:19:10.532 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:10.532 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:10.532 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:10.532 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:19:10.532 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:19:10.532 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:19:10.536 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:19:10.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:10.536 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:10.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:10.566 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:10.637 INFO:teuthology.orchestra.run.smithi123.stdout:client4588 2023-11-24T06:19:10.638 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:10.638 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:10.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:10.695 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:10.767 INFO:teuthology.orchestra.run.smithi123.stdout:client4588 2023-11-24T06:19:10.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:10.768 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:10.839 INFO:teuthology.orchestra.run.smithi123.stdout:client4588 2023-11-24T06:19:10.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:10.839 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/be2488f4-68a7-42cc-93d1-ae9c6921fb48.client4588/status') 2023-11-24T06:19:10.919 INFO:teuthology.orchestra.run.smithi123.stdout:instance: client.4588 (3)192.168.0.1:0/1082615792 2023-11-24T06:19:10.919 INFO:teuthology.orchestra.run.smithi123.stdout:blocklisted: false 2023-11-24T06:19:10.919 INFO:teuthology.orchestra.run.smithi123.stderr:0+1 records in 2023-11-24T06:19:10.920 INFO:teuthology.orchestra.run.smithi123.stderr:0+1 records out 2023-11-24T06:19:10.920 INFO:teuthology.orchestra.run.smithi123.stderr:69 bytes copied, 2.3144e-05 s, 3.0 MB/s 2023-11-24T06:19:10.921 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2023-11-24T06:19:10.921 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T06:19:10.921 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-24T06:19:10.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:19:11.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.051+0000 7f1f6063e700 1 Processor -- start 2023-11-24T06:19:11.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.051+0000 7f1f6063e700 1 -- start start 2023-11-24T06:19:11.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.052+0000 7f1f6063e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f580af4d0 0x7f1f580b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:11.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.052+0000 7f1f6063e700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580ae430 con 0x7f1f5815e080 2023-11-24T06:19:11.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.052+0000 7f1f6063e700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580ae710 con 0x7f1f5806c710 2023-11-24T06:19:11.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.052+0000 7f1f6063e700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f580af0b0 con 0x7f1f580af4d0 2023-11-24T06:19:11.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.052+0000 7f1f5ebdb700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f580af4d0 0x7f1f580b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:11.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.052+0000 7f1f5e3da700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1f5815e080 0x7f1f5806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39282/0 (socket says 172.21.15.89:39282) 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.052+0000 7f1f5e3da700 1 -- 172.21.15.89:0/113100220 learned_addr learned my addr 172.21.15.89:0/113100220 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.052+0000 7f1f5ebdb700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f580af4d0 0x7f1f580b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35782/0 (socket says 172.21.15.89:35782) 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f5ebdb700 1 -- 172.21.15.89:0/113100220 >> v1:172.21.15.123:6789/0 conn(0x7f1f5806c710 legacy=0x7f1f5806e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f5ebdb700 1 -- 172.21.15.89:0/113100220 >> v1:172.21.15.89:6789/0 conn(0x7f1f5815e080 legacy=0x7f1f5806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f5d3d8700 1 -- 172.21.15.89:0/113100220 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237310400 0 0) 0x7f1f580ae430 con 0x7f1f5815e080 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f5ebdb700 1 -- 172.21.15.89:0/113100220 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580ae9f0 con 0x7f1f580af4d0 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f5ebdb700 1 --2- 172.21.15.89:0/113100220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f580af4d0 0x7f1f580b5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009d90 tx=0x7f1f5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2591cd5c783da46 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f5d3d8700 1 -- 172.21.15.89:0/113100220 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5000e070 con 0x7f1f580af4d0 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f5d3d8700 1 -- 172.21.15.89:0/113100220 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5000bbd0 con 0x7f1f580af4d0 2023-11-24T06:19:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f5d3d8700 1 -- 172.21.15.89:0/113100220 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50012430 con 0x7f1f580af4d0 2023-11-24T06:19:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f6063e700 1 -- 172.21.15.89:0/113100220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f580af4d0 msgr2=0x7f1f580b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.053+0000 7f1f6063e700 1 --2- 172.21.15.89:0/113100220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f580af4d0 0x7f1f580b5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009d90 tx=0x7f1f5000b330 comp rx=0 tx=0).stop 2023-11-24T06:19:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.054+0000 7f1f6063e700 1 -- 172.21.15.89:0/113100220 shutdown_connections 2023-11-24T06:19:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.054+0000 7f1f6063e700 1 --2- 172.21.15.89:0/113100220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f580af4d0 0x7f1f580b5050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.054+0000 7f1f6063e700 1 -- 172.21.15.89:0/113100220 >> 172.21.15.89:0/113100220 conn(0x7f1f58071b90 msgr2=0x7f1f58071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.054+0000 7f1f6063e700 1 -- 172.21.15.89:0/113100220 shutdown_connections 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.054+0000 7f1f6063e700 1 -- 172.21.15.89:0/113100220 wait complete. 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.055+0000 7f1f6063e700 1 Processor -- start 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.055+0000 7f1f6063e700 1 -- start start 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f6063e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f5806c710 0x7f1f58153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f6063e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f580af4d0 0x7f1f581607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f6063e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f5815e080 0x7f1f58141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f6063e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581542a0 con 0x7f1f5815e080 2023-11-24T06:19:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f6063e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5815f4b0 con 0x7f1f580af4d0 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f6063e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5815f750 con 0x7f1f5806c710 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f5815e080 0x7f1f58141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f5815e080 0x7f1f58141950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35788/0 (socket says 172.21.15.89:35788) 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 -- 172.21.15.89:0/3569587050 learned_addr learned my addr 172.21.15.89:0/3569587050 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5dbd9700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f580af4d0 0x7f1f581607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5e3da700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f5806c710 0x7f1f58153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 -- 172.21.15.89:0/3569587050 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f5806c710 msgr2=0x7f1f58153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f5806c710 0x7f1f58153d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 -- 172.21.15.89:0/3569587050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f580af4d0 msgr2=0x7f1f581607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f580af4d0 0x7f1f581607f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 -- 172.21.15.89:0/3569587050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f50009a70 con 0x7f1f5815e080 2023-11-24T06:19:11.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f5ebdb700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f5815e080 0x7f1f58141950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1f50014040 tx=0x7f1f50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:11.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.056+0000 7f1f4effd700 1 -- 172.21.15.89:0/3569587050 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50012430 con 0x7f1f5815e080 2023-11-24T06:19:11.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.057+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5815fa30 con 0x7f1f5815e080 2023-11-24T06:19:11.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.057+0000 7f1f4effd700 1 -- 172.21.15.89:0/3569587050 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f50020070 con 0x7f1f5815e080 2023-11-24T06:19:11.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.057+0000 7f1f4effd700 1 -- 172.21.15.89:0/3569587050 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50004280 con 0x7f1f5815e080 2023-11-24T06:19:11.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.057+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5815fff0 con 0x7f1f5815e080 2023-11-24T06:19:11.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.058+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f580753f0 con 0x7f1f5815e080 2023-11-24T06:19:11.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.058+0000 7f1f4effd700 1 -- 172.21.15.89:0/3569587050 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1f500043e0 con 0x7f1f5815e080 2023-11-24T06:19:11.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.058+0000 7f1f4effd700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f3c06aed0 0x7f1f3c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:11.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.059+0000 7f1f4effd700 1 -- 172.21.15.89:0/3569587050 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f50022070 con 0x7f1f5815e080 2023-11-24T06:19:11.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.059+0000 7f1f5e3da700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f3c06aed0 0x7f1f3c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:11.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.059+0000 7f1f5e3da700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f3c06aed0 0x7f1f3c06d390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1f48000f60 tx=0x7f1f48004a50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:11.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.063+0000 7f1f4effd700 1 -- 172.21.15.89:0/3569587050 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f50056490 con 0x7f1f5815e080 2023-11-24T06:19:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.301+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1f58067f60 con 0x7f1f5815e080 2023-11-24T06:19:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.301+0000 7f1f4effd700 1 -- 172.21.15.89:0/3569587050 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1f50059ab0 con 0x7f1f5815e080 2023-11-24T06:19:11.305 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:19:11.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f3c06aed0 msgr2=0x7f1f3c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:11.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f3c06aed0 0x7f1f3c06d390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1f48000f60 tx=0x7f1f48004a50 comp rx=0 tx=0).stop 2023-11-24T06:19:11.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f5815e080 msgr2=0x7f1f58141950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f5815e080 0x7f1f58141950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1f50014040 tx=0x7f1f50004ad0 comp rx=0 tx=0).stop 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 shutdown_connections 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f5815e080 0x7f1f58141950 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f5806c710 0x7f1f58153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f3c06aed0 0x7f1f3c06d390 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 --2- 172.21.15.89:0/3569587050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f580af4d0 0x7f1f581607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 >> 172.21.15.89:0/3569587050 conn(0x7f1f58071b90 msgr2=0x7f1f580ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.303+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 shutdown_connections 2023-11-24T06:19:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:11.304+0000 7f1f6063e700 1 -- 172.21.15.89:0/3569587050 wait complete. 2023-11-24T06:19:11.318 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T06:19:11.318 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:19:11.319 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2023-11-24T06:19:11.320 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-11-24T06:19:11.320 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-24T06:19:11.320 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T06:19:11.320 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T06:19:11.320 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:19:11.320 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-11-24T06:19:11.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.320 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:11.338 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42400sec preferred_lft 42400sec 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::a981:dd7b:bd0d:851a/64 scope link noprefixroute 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether d2:06:87:3a:6b:fb brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:19:11.339 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.340 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::682d:1bff:fea0:f45c/64 scope link 2023-11-24T06:19:11.340 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.340 INFO:teuthology.orchestra.run.smithi123.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:19:11.340 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether d2:06:87:3a:6b:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:19:11.340 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::d006:87ff:fe3a:6bfb/64 scope link tentative 2023-11-24T06:19:11.340 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.340 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:19:11.404 INFO:teuthology.orchestra.run.smithi123.stdout:1 2023-11-24T06:19:11.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.405 DEBUG:teuthology.orchestra.run.smithi123:> route 2023-11-24T06:19:11.464 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2023-11-24T06:19:11.464 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:19:11.464 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:19:11.464 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:19:11.464 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:19:11.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.465 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:11.465 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:11.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:19:11.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:19:11.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:19:11.465 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:11.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.568 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:19:11.585 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T06:19:11.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.585 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2023-11-24T06:19:11.642 INFO:teuthology.orchestra.run.smithi123.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-11-24T06:19:11.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.642 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:11.642 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:11.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T06:19:11.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-11-24T06:19:11.643 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:11.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-24T06:19:11.780 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:11.781 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:11.781 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:11.781 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.781 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:11.781 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.781 INFO:teuthology.orchestra.run.smithi123.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:19:11.782 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 96:de:fe:a0:ca:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T06:19:11.782 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T06:19:11.782 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.782 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::94de:feff:fea0:ca43/64 scope link tentative 2023-11-24T06:19:11.782 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-24T06:19:11.866 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:11.867 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:11.867 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:11.867 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.867 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:11.867 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.867 INFO:teuthology.orchestra.run.smithi123.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:19:11.867 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 96:de:fe:a0:ca:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T06:19:11.868 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T06:19:11.868 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.868 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::94de:feff:fea0:ca43/64 scope link tentative 2023-11-24T06:19:11.868 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:11.883 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-11-24T06:19:11.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:11.883 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:11.883 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:11.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-11-24T06:19:11.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-24T06:19:11.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-11-24T06:19:11.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-11-24T06:19:11.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-11-24T06:19:11.884 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:12.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:12.137 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:12.137 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:12.137 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.1 up 2023-11-24T06:19:12.137 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.1 master ceph-brx 2023-11-24T06:19:12.137 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:12.249 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:12.249 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-11-24T06:19:12.272 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-11-24T06:19:12.272 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:12.272 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-11-24T06:19:12.328 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:19:12.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:12.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-24T06:19:12.414 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-24T06:19:12.414 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-24T06:19:12.414 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:12.422 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-24T06:19:12.422 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-24T06:19:12.422 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:12.423 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:12.423 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:12.454 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-24T06:19:12.455 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-24T06:19:12.455 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:12.455 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:12.455 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:12.455 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:19:12.455 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:19:12.455 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:19:12.459 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:19:12.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:12.459 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-11-24T06:19:12.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:12.492 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:12.563 INFO:teuthology.orchestra.run.smithi123.stdout:client4457 2023-11-24T06:19:12.564 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:12.564 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:12.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:12.622 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:12.694 INFO:teuthology.orchestra.run.smithi123.stdout:client4457 2023-11-24T06:19:12.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:12.702 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:12.764 INFO:teuthology.orchestra.run.smithi123.stdout:client4457 2023-11-24T06:19:12.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:12.765 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/be2488f4-68a7-42cc-93d1-ae9c6921fb48.client4457/status') 2023-11-24T06:19:12.844 INFO:teuthology.orchestra.run.smithi123.stdout:instance: client.4457 (3)192.168.0.2:0/1986635589 2023-11-24T06:19:12.844 INFO:teuthology.orchestra.run.smithi123.stdout:blocklisted: false 2023-11-24T06:19:12.845 INFO:teuthology.orchestra.run.smithi123.stderr:0+1 records in 2023-11-24T06:19:12.845 INFO:teuthology.orchestra.run.smithi123.stderr:0+1 records out 2023-11-24T06:19:12.846 INFO:teuthology.orchestra.run.smithi123.stderr:69 bytes copied, 2.3031e-05 s, 3.0 MB/s 2023-11-24T06:19:12.846 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-11-24T06:19:12.878 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:13.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.009+0000 7f13e1a91700 1 Processor -- start 2023-11-24T06:19:13.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.009+0000 7f13e1a91700 1 -- start start 2023-11-24T06:19:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.010+0000 7f13e1a91700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13dc0b3370 0x7f13dc1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.010+0000 7f13e1a91700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc0b22d0 con 0x7f13dc0d1c70 2023-11-24T06:19:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.010+0000 7f13e1a91700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc0b25b0 con 0x7f13dc15bdd0 2023-11-24T06:19:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.010+0000 7f13e1a91700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13dc0b2f50 con 0x7f13dc0b3370 2023-11-24T06:19:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.010+0000 7f13daffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13dc0b3370 0x7f13dc1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.010+0000 7f13dbfff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f13dc0d1c70 0x7f13dc165ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39300/0 (socket says 172.21.15.89:39300) 2023-11-24T06:19:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.010+0000 7f13dbfff700 1 -- 172.21.15.89:0/3243768076 learned_addr learned my addr 172.21.15.89:0/3243768076 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13da7fc700 1 -- 172.21.15.89:0/3243768076 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154041326 0 0) 0x7f13dc0b22d0 con 0x7f13dc0d1c70 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13da7fc700 1 -- 172.21.15.89:0/3243768076 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c40035b0 con 0x7f13dc0d1c70 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13da7fc700 1 -- 172.21.15.89:0/3243768076 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588052512 0 0) 0x7f13dc0b25b0 con 0x7f13dc15bdd0 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13da7fc700 1 -- 172.21.15.89:0/3243768076 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13dc0b22d0 con 0x7f13dc15bdd0 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13daffd700 1 -- 172.21.15.89:0/3243768076 >> v1:172.21.15.123:6789/0 conn(0x7f13dc15bdd0 legacy=0x7f13dc0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13daffd700 1 -- 172.21.15.89:0/3243768076 >> v1:172.21.15.89:6789/0 conn(0x7f13dc0d1c70 legacy=0x7f13dc165ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13daffd700 1 -- 172.21.15.89:0/3243768076 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc0b2890 con 0x7f13dc0b3370 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13daffd700 1 --2- 172.21.15.89:0/3243768076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc0b3370 0x7f13dc1604e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009840 tx=0x7f13d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769dc488852e6adf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13da7fc700 1 -- 172.21.15.89:0/3243768076 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0011830 con 0x7f13dc0b3370 2023-11-24T06:19:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13da7fc700 1 -- 172.21.15.89:0/3243768076 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0011990 con 0x7f13dc0b3370 2023-11-24T06:19:13.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.011+0000 7f13da7fc700 1 -- 172.21.15.89:0/3243768076 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0011d00 con 0x7f13dc0b3370 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.012+0000 7f13e1a91700 1 -- 172.21.15.89:0/3243768076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc0b3370 msgr2=0x7f13dc1604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.012+0000 7f13e1a91700 1 --2- 172.21.15.89:0/3243768076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc0b3370 0x7f13dc1604e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009840 tx=0x7f13d0009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.012+0000 7f13e1a91700 1 -- 172.21.15.89:0/3243768076 shutdown_connections 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.012+0000 7f13e1a91700 1 --2- 172.21.15.89:0/3243768076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc0b3370 0x7f13dc1604e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.012+0000 7f13e1a91700 1 -- 172.21.15.89:0/3243768076 >> 172.21.15.89:0/3243768076 conn(0x7f13dc0731a0 msgr2=0x7f13dc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.012+0000 7f13e1a91700 1 -- 172.21.15.89:0/3243768076 shutdown_connections 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.012+0000 7f13e1a91700 1 -- 172.21.15.89:0/3243768076 wait complete. 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.013+0000 7f13e1a91700 1 Processor -- start 2023-11-24T06:19:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.014+0000 7f13e1a91700 1 -- start start 2023-11-24T06:19:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.014+0000 7f13e1a91700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b3370 0x7f13dc14fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.014+0000 7f13e1a91700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc0d1c70 0x7f13dc0cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.014+0000 7f13e1a91700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc15bdd0 0x7f13dc0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.014+0000 7f13e1a91700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc15dc80 con 0x7f13dc15bdd0 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.014+0000 7f13e1a91700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc150590 con 0x7f13dc0b3370 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.014+0000 7f13e1a91700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc1f2db0 con 0x7f13dc0d1c70 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc15bdd0 0x7f13dc0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13db7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b3370 0x7f13dc14fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc15bdd0 0x7f13dc0ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35802/0 (socket says 172.21.15.89:35802) 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13daffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc0d1c70 0x7f13dc0cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 -- 172.21.15.89:0/4134178897 learned_addr learned my addr 172.21.15.89:0/4134178897 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 -- 172.21.15.89:0/4134178897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc0d1c70 msgr2=0x7f13dc0cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc0d1c70 0x7f13dc0cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 -- 172.21.15.89:0/4134178897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b3370 msgr2=0x7f13dc14fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b3370 0x7f13dc14fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 -- 172.21.15.89:0/4134178897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d00094a0 con 0x7f13dc15bdd0 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13db7fe700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b3370 0x7f13dc14fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13dbfff700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc15bdd0 0x7f13dc0ced50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f13d400cb40 tx=0x7f13d400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13c3fff700 1 -- 172.21.15.89:0/4134178897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d400a9e0 con 0x7f13dc15bdd0 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13c3fff700 1 -- 172.21.15.89:0/4134178897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d400ab40 con 0x7f13dc15bdd0 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13c3fff700 1 -- 172.21.15.89:0/4134178897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4012400 con 0x7f13dc15bdd0 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc1f3090 con 0x7f13dc15bdd0 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.015+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc0ce550 con 0x7f13dc15bdd0 2023-11-24T06:19:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.016+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc0ce0c0 con 0x7f13dc15bdd0 2023-11-24T06:19:13.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.018+0000 7f13c3fff700 1 -- 172.21.15.89:0/4134178897 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f13d401b3b0 con 0x7f13dc15bdd0 2023-11-24T06:19:13.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.018+0000 7f13c3fff700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f13c806aed0 0x7f13c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.018+0000 7f13c3fff700 1 -- 172.21.15.89:0/4134178897 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13d4089550 con 0x7f13dc15bdd0 2023-11-24T06:19:13.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.018+0000 7f13db7fe700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f13c806aed0 0x7f13c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.018+0000 7f13db7fe700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f13c806aed0 0x7f13c806d390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0011e0 tx=0x7f13cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.022+0000 7f13c3fff700 1 -- 172.21.15.89:0/4134178897 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13d4054770 con 0x7f13dc15bdd0 2023-11-24T06:19:13.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.213+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13dc067f60 con 0x7f13dc15bdd0 2023-11-24T06:19:13.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.214+0000 7f13c3fff700 1 -- 172.21.15.89:0/4134178897 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f13d4057d90 con 0x7f13dc15bdd0 2023-11-24T06:19:13.217 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:13.217 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f13c806aed0 msgr2=0x7f13c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f13c806aed0 0x7f13c806d390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0011e0 tx=0x7f13cc009040 comp rx=0 tx=0).stop 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc15bdd0 msgr2=0x7f13dc0ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc15bdd0 0x7f13dc0ced50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f13d400cb40 tx=0x7f13d400ce50 comp rx=0 tx=0).stop 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 shutdown_connections 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13dc15bdd0 0x7f13dc0ced50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f13c806aed0 0x7f13c806d390 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b3370 0x7f13dc14fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 --2- 172.21.15.89:0/4134178897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc0d1c70 0x7f13dc0cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 >> 172.21.15.89:0/4134178897 conn(0x7f13dc0731a0 msgr2=0x7f13dc0b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 shutdown_connections 2023-11-24T06:19:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.216+0000 7f13e1a91700 1 -- 172.21.15.89:0/4134178897 wait complete. 2023-11-24T06:19:13.232 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:13.232 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.353+0000 7f9ca37cb700 1 Processor -- start 2023-11-24T06:19:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.353+0000 7f9ca37cb700 1 -- start start 2023-11-24T06:19:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca37cb700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9c9c1587f0 0x7f9c9c06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca37cb700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c9c0b4370 0x7f9c9c15b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca37cb700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c06b530 con 0x7f9c9c15b370 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca37cb700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c0ae5e0 con 0x7f9c9c0b4370 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca37cb700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c15be30 con 0x7f9c9c1587f0 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1d68700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c9c0b4370 0x7f9c9c15b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1d68700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c9c0b4370 0x7f9c9c15b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35810/0 (socket says 172.21.15.89:35810) 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1d68700 1 -- 172.21.15.89:0/3253533555 learned_addr learned my addr 172.21.15.89:0/3253533555 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1567700 1 --2- 172.21.15.89:0/3253533555 >> v2:172.21.15.123:3300/0 conn(0x7f9c9c1587f0 0x7f9c9c06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1d68700 1 -- 172.21.15.89:0/3253533555 >> v2:172.21.15.123:3300/0 conn(0x7f9c9c1587f0 msgr2=0x7f9c9c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1d68700 1 --2- 172.21.15.89:0/3253533555 >> v2:172.21.15.123:3300/0 conn(0x7f9c9c1587f0 0x7f9c9c06aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1d68700 1 -- 172.21.15.89:0/3253533555 >> v1:172.21.15.123:6789/0 conn(0x7f9c9c15b370 legacy=0x7f9c9c0b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1d68700 1 -- 172.21.15.89:0/3253533555 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c15c110 con 0x7f9c9c0b4370 2023-11-24T06:19:13.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9ca1d68700 1 --2- 172.21.15.89:0/3253533555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c0b4370 0x7f9c9c15b800 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400d150 tx=0x7f9c9400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8078b71868988442 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9c93fff700 1 -- 172.21.15.89:0/3253533555 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94010070 con 0x7f9c9c0b4370 2023-11-24T06:19:13.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9c93fff700 1 -- 172.21.15.89:0/3253533555 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9400dcd0 con 0x7f9c9c0b4370 2023-11-24T06:19:13.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.354+0000 7f9c93fff700 1 -- 172.21.15.89:0/3253533555 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9400b310 con 0x7f9c9c0b4370 2023-11-24T06:19:13.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.355+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3253533555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c0b4370 msgr2=0x7f9c9c15b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.355+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3253533555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c0b4370 0x7f9c9c15b800 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400d150 tx=0x7f9c9400d460 comp rx=0 tx=0).stop 2023-11-24T06:19:13.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.355+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3253533555 shutdown_connections 2023-11-24T06:19:13.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.355+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3253533555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c0b4370 0x7f9c9c15b800 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.355+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3253533555 >> v2:172.21.15.123:3300/0 conn(0x7f9c9c1587f0 0x7f9c9c06aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.355+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3253533555 >> 172.21.15.89:0/3253533555 conn(0x7f9c9c0d0bf0 msgr2=0x7f9c9c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.355+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3253533555 shutdown_connections 2023-11-24T06:19:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.355+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3253533555 wait complete. 2023-11-24T06:19:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.356+0000 7f9ca37cb700 1 Processor -- start 2023-11-24T06:19:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.356+0000 7f9ca37cb700 1 -- start start 2023-11-24T06:19:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca37cb700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c9c0b4370 0x7f9c9c150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca37cb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1587f0 0x7f9c9c06b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca37cb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15b370 0x7f9c9c06bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca37cb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c1f2e30 con 0x7f9c9c1587f0 2023-11-24T06:19:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca37cb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c154070 con 0x7f9c9c15b370 2023-11-24T06:19:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca37cb700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c1f3080 con 0x7f9c9c0b4370 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1587f0 0x7f9c9c06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1587f0 0x7f9c9c06b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35812/0 (socket says 172.21.15.89:35812) 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 -- 172.21.15.89:0/3932710210 learned_addr learned my addr 172.21.15.89:0/3932710210 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca1567700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c9c0b4370 0x7f9c9c150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca1d68700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15b370 0x7f9c9c06bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 -- 172.21.15.89:0/3932710210 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c9c0b4370 msgr2=0x7f9c9c150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c9c0b4370 0x7f9c9c150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 -- 172.21.15.89:0/3932710210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15b370 msgr2=0x7f9c9c06bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15b370 0x7f9c9c06bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 -- 172.21.15.89:0/3932710210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9400bd40 con 0x7f9c9c1587f0 2023-11-24T06:19:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.357+0000 7f9ca0d66700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1587f0 0x7f9c9c06b960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c00c450 tx=0x7f9c8c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.358+0000 7f9c91ffb700 1 -- 172.21.15.89:0/3932710210 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0052d0 con 0x7f9c9c1587f0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.358+0000 7f9c91ffb700 1 -- 172.21.15.89:0/3932710210 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8c005430 con 0x7f9c9c1587f0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.358+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c1f3360 con 0x7f9c9c1587f0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.358+0000 7f9c91ffb700 1 -- 172.21.15.89:0/3932710210 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0038c0 con 0x7f9c9c1587f0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.358+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c150bb0 con 0x7f9c9c1587f0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.359+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c068fb0 con 0x7f9c9c1587f0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.359+0000 7f9c91ffb700 1 -- 172.21.15.89:0/3932710210 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9c8c003a20 con 0x7f9c9c1587f0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.359+0000 7f9c91ffb700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9c8006aed0 0x7f9c8006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.359+0000 7f9c91ffb700 1 -- 172.21.15.89:0/3932710210 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c8c088830 con 0x7f9c9c1587f0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.359+0000 7f9ca1567700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9c8006aed0 0x7f9c8006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.360+0000 7f9ca1567700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9c8006aed0 0x7f9c8006d390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009be0 tx=0x7f9c98009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.364+0000 7f9c91ffb700 1 -- 172.21.15.89:0/3932710210 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c8c053970 con 0x7f9c9c1587f0 2023-11-24T06:19:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.554+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c9c1f3820 con 0x7f9c9c1587f0 2023-11-24T06:19:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.554+0000 7f9c91ffb700 1 -- 172.21.15.89:0/3932710210 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c8c0170a0 con 0x7f9c9c1587f0 2023-11-24T06:19:13.557 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:13.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9c8006aed0 msgr2=0x7f9c8006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9c8006aed0 0x7f9c8006d390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009be0 tx=0x7f9c98009210 comp rx=0 tx=0).stop 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1587f0 msgr2=0x7f9c9c06b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1587f0 0x7f9c9c06b960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c00c450 tx=0x7f9c8c00c760 comp rx=0 tx=0).stop 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 shutdown_connections 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9c8006aed0 0x7f9c8006d390 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15b370 0x7f9c9c06bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c9c0b4370 0x7f9c9c150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 --2- 172.21.15.89:0/3932710210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1587f0 0x7f9c9c06b960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 >> 172.21.15.89:0/3932710210 conn(0x7f9c9c0d0bf0 msgr2=0x7f9c9c15de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:13.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 shutdown_connections 2023-11-24T06:19:13.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.556+0000 7f9ca37cb700 1 -- 172.21.15.89:0/3932710210 wait complete. 2023-11-24T06:19:13.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:13.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.692+0000 7fb431706700 1 Processor -- start 2023-11-24T06:19:13.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.692+0000 7fb431706700 1 -- start start 2023-11-24T06:19:13.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.693+0000 7fb431706700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c15e160 0x7fb42c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.693+0000 7fb431706700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb42c0afc90 0x7fb42c073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.693+0000 7fb431706700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c0b3c60 con 0x7fb42c0744d0 2023-11-24T06:19:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.693+0000 7fb431706700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c0b4500 con 0x7fb42c0afc90 2023-11-24T06:19:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.693+0000 7fb431706700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c074960 con 0x7fb42c15e160 2023-11-24T06:19:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.693+0000 7fb42b7fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb42c0744d0 0x7fb42c06d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39328/0 (socket says 172.21.15.89:39328) 2023-11-24T06:19:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.693+0000 7fb42a7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb42c0afc90 0x7fb42c073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.693+0000 7fb42b7fe700 1 -- 172.21.15.89:0/3289832160 learned_addr learned my addr 172.21.15.89:0/3289832160 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb42affd700 1 --2- 172.21.15.89:0/3289832160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c15e160 0x7fb42c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb429ffb700 1 -- 172.21.15.89:0/3289832160 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643025210 0 0) 0x7fb42c0b3c60 con 0x7fb42c0744d0 2023-11-24T06:19:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb429ffb700 1 -- 172.21.15.89:0/3289832160 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4140035b0 con 0x7fb42c0744d0 2023-11-24T06:19:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb42a7fc700 1 -- 172.21.15.89:0/3289832160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c15e160 msgr2=0x7fb42c0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb42a7fc700 1 --2- 172.21.15.89:0/3289832160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c15e160 0x7fb42c0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb42a7fc700 1 -- 172.21.15.89:0/3289832160 >> v1:172.21.15.89:6789/0 conn(0x7fb42c0744d0 legacy=0x7fb42c06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb42a7fc700 1 -- 172.21.15.89:0/3289832160 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c0b58c0 con 0x7fb42c0afc90 2023-11-24T06:19:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb42a7fc700 1 --2- 172.21.15.89:0/3289832160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c073f90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009840 tx=0x7fb41c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97bfc7b100cb801 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb429ffb700 1 -- 172.21.15.89:0/3289832160 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c011830 con 0x7fb42c0afc90 2023-11-24T06:19:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb429ffb700 1 -- 172.21.15.89:0/3289832160 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c011990 con 0x7fb42c0afc90 2023-11-24T06:19:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb429ffb700 1 -- 172.21.15.89:0/3289832160 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c011d20 con 0x7fb42c0afc90 2023-11-24T06:19:13.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb431706700 1 -- 172.21.15.89:0/3289832160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 msgr2=0x7fb42c073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.694+0000 7fb431706700 1 --2- 172.21.15.89:0/3289832160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c073f90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009840 tx=0x7fb41c009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:13.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.695+0000 7fb431706700 1 -- 172.21.15.89:0/3289832160 shutdown_connections 2023-11-24T06:19:13.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.695+0000 7fb431706700 1 --2- 172.21.15.89:0/3289832160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c15e160 0x7fb42c0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.695+0000 7fb431706700 1 --2- 172.21.15.89:0/3289832160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c073f90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.695+0000 7fb431706700 1 -- 172.21.15.89:0/3289832160 >> 172.21.15.89:0/3289832160 conn(0x7fb42c0d0bf0 msgr2=0x7fb42c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.695+0000 7fb431706700 1 -- 172.21.15.89:0/3289832160 shutdown_connections 2023-11-24T06:19:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.695+0000 7fb431706700 1 -- 172.21.15.89:0/3289832160 wait complete. 2023-11-24T06:19:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.696+0000 7fb431706700 1 Processor -- start 2023-11-24T06:19:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.696+0000 7fb431706700 1 -- start start 2023-11-24T06:19:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb431706700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c0744d0 0x7fb42c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb431706700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c0add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb431706700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c15e160 0x7fb42c0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb431706700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c14cb60 con 0x7fb42c0afc90 2023-11-24T06:19:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb431706700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c14c530 con 0x7fb42c15e160 2023-11-24T06:19:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb431706700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1f7830 con 0x7fb42c0744d0 2023-11-24T06:19:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c0add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c0add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35828/0 (socket says 172.21.15.89:35828) 2023-11-24T06:19:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 -- 172.21.15.89:0/4059393220 learned_addr learned my addr 172.21.15.89:0/4059393220 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42affd700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c0744d0 0x7fb42c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42b7fe700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c15e160 0x7fb42c0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 -- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c0744d0 msgr2=0x7fb42c0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c0744d0 0x7fb42c0c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 -- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c15e160 msgr2=0x7fb42c0ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c15e160 0x7fb42c0ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 -- 172.21.15.89:0/4059393220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c0094a0 con 0x7fb42c0afc90 2023-11-24T06:19:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42affd700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c0744d0 0x7fb42c0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42b7fe700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c15e160 0x7fb42c0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:13.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb42a7fc700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c0add40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb41c000c00 tx=0x7fb41c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb4137fe700 1 -- 172.21.15.89:0/4059393220 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0036a0 con 0x7fb42c0afc90 2023-11-24T06:19:13.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb4137fe700 1 -- 172.21.15.89:0/4059393220 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c01f070 con 0x7fb42c0afc90 2023-11-24T06:19:13.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c14c7f0 con 0x7fb42c0afc90 2023-11-24T06:19:13.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb4137fe700 1 -- 172.21.15.89:0/4059393220 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c003a40 con 0x7fb42c0afc90 2023-11-24T06:19:13.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.697+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c14cff0 con 0x7fb42c0afc90 2023-11-24T06:19:13.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.698+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c14cd10 con 0x7fb42c0afc90 2023-11-24T06:19:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.699+0000 7fb4137fe700 1 -- 172.21.15.89:0/4059393220 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb41c003ba0 con 0x7fb42c0afc90 2023-11-24T06:19:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.699+0000 7fb4137fe700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb41806aed0 0x7fb41806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.699+0000 7fb4137fe700 1 -- 172.21.15.89:0/4059393220 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb41c021070 con 0x7fb42c0afc90 2023-11-24T06:19:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.699+0000 7fb42affd700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb41806aed0 0x7fb41806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.700+0000 7fb42affd700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb41806aed0 0x7fb41806d390 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb420009ac0 tx=0x7fb420008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:13.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.704+0000 7fb4137fe700 1 -- 172.21.15.89:0/4059393220 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb42c14cd10 con 0x7fb42c0afc90 2023-11-24T06:19:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.894+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb42c068fb0 con 0x7fb42c0afc90 2023-11-24T06:19:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.895+0000 7fb4137fe700 1 -- 172.21.15.89:0/4059393220 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb41c0150a0 con 0x7fb42c0afc90 2023-11-24T06:19:13.898 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:13.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb41806aed0 msgr2=0x7fb41806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb41806aed0 0x7fb41806d390 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb420009ac0 tx=0x7fb420008040 comp rx=0 tx=0).stop 2023-11-24T06:19:13.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 msgr2=0x7fb42c0add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:13.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c0add40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb41c000c00 tx=0x7fb41c004750 comp rx=0 tx=0).stop 2023-11-24T06:19:13.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 shutdown_connections 2023-11-24T06:19:13.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb41806aed0 0x7fb41806d390 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c15e160 0x7fb42c0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb42c0744d0 0x7fb42c0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 --2- 172.21.15.89:0/4059393220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb42c0afc90 0x7fb42c0add40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:13.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 >> 172.21.15.89:0/4059393220 conn(0x7fb42c0d0bf0 msgr2=0x7fb42c15bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:13.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 shutdown_connections 2023-11-24T06:19:13.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:13.897+0000 7fb431706700 1 -- 172.21.15.89:0/4059393220 wait complete. 2023-11-24T06:19:13.912 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.035+0000 7fb1a6468700 1 Processor -- start 2023-11-24T06:19:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.035+0000 7fb1a6468700 1 -- start start 2023-11-24T06:19:14.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb1a6468700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1a00d45d0 0x7fb1a00b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb1a6468700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb1a00d39d0 0x7fb1a00b7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb1a6468700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a00d2930 con 0x7fb1a015e140 2023-11-24T06:19:14.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb1a6468700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a015e520 con 0x7fb1a00d45d0 2023-11-24T06:19:14.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb1a6468700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a00d2c10 con 0x7fb1a00d39d0 2023-11-24T06:19:14.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb19f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1a00d45d0 0x7fb1a00b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb19f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1a00d45d0 0x7fb1a00b70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35836/0 (socket says 172.21.15.89:35836) 2023-11-24T06:19:14.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb19ffff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb1a015e140 0x7fb1a00d3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39338/0 (socket says 172.21.15.89:39338) 2023-11-24T06:19:14.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb19f7fe700 1 -- 172.21.15.89:0/595362759 learned_addr learned my addr 172.21.15.89:0/595362759 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:14.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb1a4a05700 1 --2- 172.21.15.89:0/595362759 >> v2:172.21.15.123:3300/0 conn(0x7fb1a00d39d0 0x7fb1a00b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.036+0000 7fb19effd700 1 -- 172.21.15.89:0/595362759 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019618912 0 0) 0x7fb1a00d2930 con 0x7fb1a015e140 2023-11-24T06:19:14.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.037+0000 7fb1a4a05700 1 -- 172.21.15.89:0/595362759 >> v2:172.21.15.89:3300/0 conn(0x7fb1a00d45d0 msgr2=0x7fb1a00b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:14.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.037+0000 7fb1a4a05700 1 --2- 172.21.15.89:0/595362759 >> v2:172.21.15.89:3300/0 conn(0x7fb1a00d45d0 0x7fb1a00b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.037+0000 7fb1a4a05700 1 -- 172.21.15.89:0/595362759 >> v1:172.21.15.89:6789/0 conn(0x7fb1a015e140 legacy=0x7fb1a00d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.037+0000 7fb1a4a05700 1 -- 172.21.15.89:0/595362759 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a00d2ef0 con 0x7fb1a00d39d0 2023-11-24T06:19:14.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.037+0000 7fb1a4a05700 1 --2- 172.21.15.89:0/595362759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 0x7fb1a00b7610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00b6df0 tx=0x7fb19800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7669aa48ccb055cf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.037+0000 7fb19effd700 1 -- 172.21.15.89:0/595362759 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19800e070 con 0x7fb1a00d39d0 2023-11-24T06:19:14.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.037+0000 7fb19effd700 1 -- 172.21.15.89:0/595362759 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19800bb20 con 0x7fb1a00d39d0 2023-11-24T06:19:14.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.037+0000 7fb19effd700 1 -- 172.21.15.89:0/595362759 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb198012430 con 0x7fb1a00d39d0 2023-11-24T06:19:14.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.038+0000 7fb1a6468700 1 -- 172.21.15.89:0/595362759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 msgr2=0x7fb1a00b7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.038+0000 7fb1a6468700 1 --2- 172.21.15.89:0/595362759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 0x7fb1a00b7610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00b6df0 tx=0x7fb19800b330 comp rx=0 tx=0).stop 2023-11-24T06:19:14.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.038+0000 7fb1a6468700 1 -- 172.21.15.89:0/595362759 shutdown_connections 2023-11-24T06:19:14.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.038+0000 7fb1a6468700 1 --2- 172.21.15.89:0/595362759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 0x7fb1a00b7610 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.038+0000 7fb1a6468700 1 --2- 172.21.15.89:0/595362759 >> v2:172.21.15.89:3300/0 conn(0x7fb1a00d45d0 0x7fb1a00b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.038+0000 7fb1a6468700 1 -- 172.21.15.89:0/595362759 >> 172.21.15.89:0/595362759 conn(0x7fb1a006b410 msgr2=0x7fb1a006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:14.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.038+0000 7fb1a6468700 1 -- 172.21.15.89:0/595362759 shutdown_connections 2023-11-24T06:19:14.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.038+0000 7fb1a6468700 1 -- 172.21.15.89:0/595362759 wait complete. 2023-11-24T06:19:14.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.039+0000 7fb1a6468700 1 Processor -- start 2023-11-24T06:19:14.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.039+0000 7fb1a6468700 1 -- start start 2023-11-24T06:19:14.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a6468700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 0x7fb1a01f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a6468700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a00d45d0 0x7fb1a0151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a6468700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a015e140 0x7fb1a00cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a6468700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a00cc500 con 0x7fb1a015e140 2023-11-24T06:19:14.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a6468700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a01f4190 con 0x7fb1a00d39d0 2023-11-24T06:19:14.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a6468700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a01f4470 con 0x7fb1a00d45d0 2023-11-24T06:19:14.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a4a05700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a015e140 0x7fb1a00cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a4a05700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a015e140 0x7fb1a00cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35844/0 (socket says 172.21.15.89:35844) 2023-11-24T06:19:14.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a4a05700 1 -- 172.21.15.89:0/1517585680 learned_addr learned my addr 172.21.15.89:0/1517585680 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:14.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb19ffff700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 0x7fb1a01f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb19f7fe700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a00d45d0 0x7fb1a0151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a4a05700 1 -- 172.21.15.89:0/1517585680 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a00d45d0 msgr2=0x7fb1a0151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a4a05700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a00d45d0 0x7fb1a0151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a4a05700 1 -- 172.21.15.89:0/1517585680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 msgr2=0x7fb1a01f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a4a05700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 0x7fb1a01f4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.040+0000 7fb1a4a05700 1 -- 172.21.15.89:0/1517585680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb198009a70 con 0x7fb1a015e140 2023-11-24T06:19:14.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.041+0000 7fb1a4a05700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a015e140 0x7fb1a00cbb60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb198014040 tx=0x7fb198004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.041+0000 7fb19cff9700 1 -- 172.21.15.89:0/1517585680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb198012430 con 0x7fb1a015e140 2023-11-24T06:19:14.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.041+0000 7fb19cff9700 1 -- 172.21.15.89:0/1517585680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb198020070 con 0x7fb1a015e140 2023-11-24T06:19:14.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.041+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a01f4750 con 0x7fb1a015e140 2023-11-24T06:19:14.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.041+0000 7fb19cff9700 1 -- 172.21.15.89:0/1517585680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb198004280 con 0x7fb1a015e140 2023-11-24T06:19:14.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.041+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a015a780 con 0x7fb1a015e140 2023-11-24T06:19:14.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.042+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a015a970 con 0x7fb1a015e140 2023-11-24T06:19:14.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.042+0000 7fb19cff9700 1 -- 172.21.15.89:0/1517585680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb1980043e0 con 0x7fb1a015e140 2023-11-24T06:19:14.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.043+0000 7fb19cff9700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18c06af00 0x7fb18c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.043+0000 7fb19cff9700 1 -- 172.21.15.89:0/1517585680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb198022070 con 0x7fb1a015e140 2023-11-24T06:19:14.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.043+0000 7fb19ffff700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18c06af00 0x7fb18c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.043+0000 7fb19ffff700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18c06af00 0x7fb18c06d3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb194000f60 tx=0x7fb194007c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.047+0000 7fb19cff9700 1 -- 172.21.15.89:0/1517585680 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb198056780 con 0x7fb1a015e140 2023-11-24T06:19:14.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.239+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb1a0068fb0 con 0x7fb1a015e140 2023-11-24T06:19:14.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.239+0000 7fb19cff9700 1 -- 172.21.15.89:0/1517585680 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb198059da0 con 0x7fb1a015e140 2023-11-24T06:19:14.242 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:14.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18c06af00 msgr2=0x7fb18c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18c06af00 0x7fb18c06d3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb194000f60 tx=0x7fb194007c90 comp rx=0 tx=0).stop 2023-11-24T06:19:14.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a015e140 msgr2=0x7fb1a00cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a015e140 0x7fb1a00cbb60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb198014040 tx=0x7fb198004ad0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 shutdown_connections 2023-11-24T06:19:14.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a015e140 0x7fb1a00cbb60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb18c06af00 0x7fb18c06d3c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a00d39d0 0x7fb1a01f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 --2- 172.21.15.89:0/1517585680 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a00d45d0 0x7fb1a0151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 >> 172.21.15.89:0/1517585680 conn(0x7fb1a006b410 msgr2=0x7fb1a006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:14.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 shutdown_connections 2023-11-24T06:19:14.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.241+0000 7fb1a6468700 1 -- 172.21.15.89:0/1517585680 wait complete. 2023-11-24T06:19:14.256 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.373+0000 7f623d9fc700 1 Processor -- start 2023-11-24T06:19:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.373+0000 7f623d9fc700 1 -- start start 2023-11-24T06:19:14.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f623d9fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f623815e1e0 0x7f62380d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f623d9fc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f62380d3a00 0x7f62380d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f623d9fc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62380d1200 con 0x7f62380d0db0 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f623d9fc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62380d3150 con 0x7f623815e1e0 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f623d9fc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623815e5c0 con 0x7f62380d3a00 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f6236ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f623815e1e0 0x7f62380d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f6236ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f623815e1e0 0x7f62380d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35848/0 (socket says 172.21.15.89:35848) 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f6236ffd700 1 -- 172.21.15.89:0/37405627 learned_addr learned my addr 172.21.15.89:0/37405627 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f62367fc700 1 --2- 172.21.15.89:0/37405627 >> v2:172.21.15.123:3300/0 conn(0x7f62380d3a00 0x7f62380d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f6236ffd700 1 -- 172.21.15.89:0/37405627 >> v2:172.21.15.123:3300/0 conn(0x7f62380d3a00 msgr2=0x7f62380d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f6236ffd700 1 --2- 172.21.15.89:0/37405627 >> v2:172.21.15.123:3300/0 conn(0x7f62380d3a00 0x7f62380d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f6236ffd700 1 -- 172.21.15.89:0/37405627 >> v1:172.21.15.123:6789/0 conn(0x7f62380d0db0 legacy=0x7f623806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.374+0000 7f6236ffd700 1 -- 172.21.15.89:0/37405627 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62380d20b0 con 0x7f623815e1e0 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f6236ffd700 1 --2- 172.21.15.89:0/37405627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f623815e1e0 0x7f62380d34c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6238161ba0 tx=0x7f6220009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd7ba41f8af02c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f6235ffb700 1 -- 172.21.15.89:0/37405627 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622000b870 con 0x7f623815e1e0 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f6235ffb700 1 -- 172.21.15.89:0/37405627 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622000b9d0 con 0x7f623815e1e0 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f6235ffb700 1 -- 172.21.15.89:0/37405627 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622000bd60 con 0x7f623815e1e0 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f623d9fc700 1 -- 172.21.15.89:0/37405627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f623815e1e0 msgr2=0x7f62380d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f623d9fc700 1 --2- 172.21.15.89:0/37405627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f623815e1e0 0x7f62380d34c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6238161ba0 tx=0x7f6220009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f623d9fc700 1 -- 172.21.15.89:0/37405627 shutdown_connections 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f623d9fc700 1 --2- 172.21.15.89:0/37405627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f623815e1e0 0x7f62380d34c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f623d9fc700 1 --2- 172.21.15.89:0/37405627 >> v2:172.21.15.123:3300/0 conn(0x7f62380d3a00 0x7f62380d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f623d9fc700 1 -- 172.21.15.89:0/37405627 >> 172.21.15.89:0/37405627 conn(0x7f62380b5b50 msgr2=0x7f62380b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f623d9fc700 1 -- 172.21.15.89:0/37405627 shutdown_connections 2023-11-24T06:19:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.375+0000 7f623d9fc700 1 -- 172.21.15.89:0/37405627 wait complete. 2023-11-24T06:19:14.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.376+0000 7f623d9fc700 1 Processor -- start 2023-11-24T06:19:14.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.377+0000 7f623d9fc700 1 -- start start 2023-11-24T06:19:14.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.377+0000 7f623d9fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62380d0db0 0x7f6238152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.377+0000 7f623d9fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380d3a00 0x7f62381f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.377+0000 7f623d9fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f623815e1e0 0x7f62381f73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.377+0000 7f623d9fc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62380cadd0 con 0x7f62380d3a00 2023-11-24T06:19:14.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.377+0000 7f623d9fc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62380cc8c0 con 0x7f62380d0db0 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.377+0000 7f623d9fc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623814c3c0 con 0x7f623815e1e0 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380d3a00 0x7f62381f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380d3a00 0x7f62381f6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35856/0 (socket says 172.21.15.89:35856) 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 -- 172.21.15.89:0/660588104 learned_addr learned my addr 172.21.15.89:0/660588104 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f6236ffd700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62380d0db0 0x7f6238152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62377fe700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f623815e1e0 0x7f62381f73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 -- 172.21.15.89:0/660588104 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f623815e1e0 msgr2=0x7f62381f73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f623815e1e0 0x7f62381f73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 -- 172.21.15.89:0/660588104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62380d0db0 msgr2=0x7f6238152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62380d0db0 0x7f6238152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 -- 172.21.15.89:0/660588104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6220009550 con 0x7f62380d3a00 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f62367fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380d3a00 0x7f62381f6e80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f622800e810 tx=0x7f622800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f622ffff700 1 -- 172.21.15.89:0/660588104 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622800c070 con 0x7f62380d3a00 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f622ffff700 1 -- 172.21.15.89:0/660588104 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6228013610 con 0x7f62380d3a00 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623814c6a0 con 0x7f62380d3a00 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f622ffff700 1 -- 172.21.15.89:0/660588104 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62280139a0 con 0x7f62380d3a00 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.378+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623814cc60 con 0x7f62380d3a00 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.379+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623814ce20 con 0x7f62380d3a00 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.380+0000 7f622ffff700 1 -- 172.21.15.89:0/660588104 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6228013b00 con 0x7f62380d3a00 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.380+0000 7f622ffff700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f621806aed0 0x7f621806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.380+0000 7f622ffff700 1 -- 172.21.15.89:0/660588104 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6228088fb0 con 0x7f62380d3a00 2023-11-24T06:19:14.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.380+0000 7f6236ffd700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f621806aed0 0x7f621806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.381+0000 7f6236ffd700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f621806aed0 0x7f621806d390 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6220014040 tx=0x7f6220008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.384+0000 7f622ffff700 1 -- 172.21.15.89:0/660588104 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62280540f0 con 0x7f62380d3a00 2023-11-24T06:19:14.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.574+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62380cca80 con 0x7f62380d3a00 2023-11-24T06:19:14.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.576+0000 7f622ffff700 1 -- 172.21.15.89:0/660588104 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f6228013db0 con 0x7f62380d3a00 2023-11-24T06:19:14.579 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:14.579 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:14.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f621806aed0 msgr2=0x7f621806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f621806aed0 0x7f621806d390 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6220014040 tx=0x7f6220008030 comp rx=0 tx=0).stop 2023-11-24T06:19:14.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380d3a00 msgr2=0x7f62381f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380d3a00 0x7f62381f6e80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f622800e810 tx=0x7f622800eb20 comp rx=0 tx=0).stop 2023-11-24T06:19:14.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 shutdown_connections 2023-11-24T06:19:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f623815e1e0 0x7f62381f73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f621806aed0 0x7f621806d390 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62380d0db0 0x7f6238152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 --2- 172.21.15.89:0/660588104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380d3a00 0x7f62381f6e80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 >> 172.21.15.89:0/660588104 conn(0x7f62380b5b50 msgr2=0x7f62380ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 shutdown_connections 2023-11-24T06:19:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.578+0000 7f623d9fc700 1 -- 172.21.15.89:0/660588104 wait complete. 2023-11-24T06:19:14.593 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:14.593 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:14.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.713+0000 7f0462f6d700 1 Processor -- start 2023-11-24T06:19:14.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.713+0000 7f0462f6d700 1 -- start start 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.714+0000 7f0462f6d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.714+0000 7f0462f6d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c0b3c60 con 0x7f045c0afc90 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.714+0000 7f0462f6d700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c0b58c0 con 0x7f045c15e160 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.714+0000 7f0462f6d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045c0b4500 con 0x7f045c0746a0 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.714+0000 7f046150a700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f045c0afc90 0x7f045c06d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39370/0 (socket says 172.21.15.89:39370) 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.714+0000 7f046150a700 1 -- 172.21.15.89:0/3798966194 learned_addr learned my addr 172.21.15.89:0/3798966194 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045bfff700 1 --2- 172.21.15.89:0/3798966194 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045b7fe700 1 -- 172.21.15.89:0/3798966194 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653683079 0 0) 0x7f045c0b3c60 con 0x7f045c0afc90 2023-11-24T06:19:14.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045b7fe700 1 -- 172.21.15.89:0/3798966194 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f044c0035b0 con 0x7f045c0afc90 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045b7fe700 1 -- 172.21.15.89:0/3798966194 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2096132600 0 0) 0x7f044c0035b0 con 0x7f045c0afc90 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045bfff700 1 -- 172.21.15.89:0/3798966194 >> v1:172.21.15.123:6789/0 conn(0x7f045c15e160 legacy=0x7f045c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045bfff700 1 -- 172.21.15.89:0/3798966194 >> v1:172.21.15.89:6789/0 conn(0x7f045c0afc90 legacy=0x7f045c06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045bfff700 1 -- 172.21.15.89:0/3798966194 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c0b0140 con 0x7f045c0746a0 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045bfff700 1 --2- 172.21.15.89:0/3798966194 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c0b5da0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0448009870 tx=0x7f0448009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=285bdd6c2cdcf1ea server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045b7fe700 1 -- 172.21.15.89:0/3798966194 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04540019f0 con 0x7f045c0746a0 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045b7fe700 1 -- 172.21.15.89:0/3798966194 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0448004030 con 0x7f045c0746a0 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.715+0000 7f045b7fe700 1 -- 172.21.15.89:0/3798966194 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04480043c0 con 0x7f045c0746a0 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.716+0000 7f0462f6d700 1 -- 172.21.15.89:0/3798966194 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 msgr2=0x7f045c0b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.716+0000 7f0462f6d700 1 --2- 172.21.15.89:0/3798966194 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c0b5da0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0448009870 tx=0x7f0448009b80 comp rx=0 tx=0).stop 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.716+0000 7f0462f6d700 1 -- 172.21.15.89:0/3798966194 shutdown_connections 2023-11-24T06:19:14.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.716+0000 7f0462f6d700 1 --2- 172.21.15.89:0/3798966194 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c0b5da0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.716+0000 7f0462f6d700 1 -- 172.21.15.89:0/3798966194 >> 172.21.15.89:0/3798966194 conn(0x7f045c0d0bf0 msgr2=0x7f045c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:14.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.716+0000 7f0462f6d700 1 -- 172.21.15.89:0/3798966194 shutdown_connections 2023-11-24T06:19:14.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.716+0000 7f0462f6d700 1 -- 172.21.15.89:0/3798966194 wait complete. 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.717+0000 7f0462f6d700 1 Processor -- start 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.717+0000 7f0462f6d700 1 -- start start 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.718+0000 7f0462f6d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.718+0000 7f0462f6d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f045c0afc90 0x7f045c14a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f0462f6d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045c15e160 0x7f045c15c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f0462f6d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c1509f0 con 0x7f045c15e160 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f0462f6d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c14a8b0 con 0x7f045c0afc90 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f0462f6d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045c14abe0 con 0x7f045c0746a0 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045c15e160 0x7f045c15c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045c15e160 0x7f045c15c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35872/0 (socket says 172.21.15.89:35872) 2023-11-24T06:19:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 -- 172.21.15.89:0/1580917010 learned_addr learned my addr 172.21.15.89:0/1580917010 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f045bfff700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f045c0afc90 0x7f045c14a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f0460d09700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 -- 172.21.15.89:0/1580917010 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 msgr2=0x7f045c150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 -- 172.21.15.89:0/1580917010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f045c0afc90 msgr2=0x7f045c14a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f045c0afc90 0x7f045c14a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 -- 172.21.15.89:0/1580917010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04480094a0 con 0x7f045c15e160 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f046150a700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045c15e160 0x7f045c15c0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f04540018d0 tx=0x7f045400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f04597fa700 1 -- 172.21.15.89:0/1580917010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045400ad40 con 0x7f045c15e160 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f04597fa700 1 -- 172.21.15.89:0/1580917010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04540070d0 con 0x7f045c15e160 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045c1f7870 con 0x7f045c15e160 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f04597fa700 1 -- 172.21.15.89:0/1580917010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454007480 con 0x7f045c15e160 2023-11-24T06:19:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.719+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045c1f7e30 con 0x7f045c15e160 2023-11-24T06:19:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.720+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045c068fb0 con 0x7f045c15e160 2023-11-24T06:19:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.721+0000 7f04597fa700 1 -- 172.21.15.89:0/1580917010 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f04540075e0 con 0x7f045c15e160 2023-11-24T06:19:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.721+0000 7f04597fa700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f044006aed0 0x7f044006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:14.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.721+0000 7f04597fa700 1 -- 172.21.15.89:0/1580917010 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0454089650 con 0x7f045c15e160 2023-11-24T06:19:14.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.721+0000 7f0460d09700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f044006aed0 0x7f044006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:14.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.721+0000 7f0460d09700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f044006aed0 0x7f044006d390 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0450000f60 tx=0x7f0450009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.725+0000 7f04597fa700 1 -- 172.21.15.89:0/1580917010 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0454054700 con 0x7f045c15e160 2023-11-24T06:19:14.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.913+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f045c14ad20 con 0x7f045c15e160 2023-11-24T06:19:14.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.913+0000 7f04597fa700 1 -- 172.21.15.89:0/1580917010 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0454057d20 con 0x7f045c15e160 2023-11-24T06:19:14.916 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f044006aed0 msgr2=0x7f044006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f044006aed0 0x7f044006d390 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0450000f60 tx=0x7f0450009040 comp rx=0 tx=0).stop 2023-11-24T06:19:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045c15e160 msgr2=0x7f045c15c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045c15e160 0x7f045c15c0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f04540018d0 tx=0x7f045400a340 comp rx=0 tx=0).stop 2023-11-24T06:19:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 shutdown_connections 2023-11-24T06:19:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045c15e160 0x7f045c15c0d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f045c0746a0 0x7f045c150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f044006aed0 0x7f044006d390 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 --2- 172.21.15.89:0/1580917010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f045c0afc90 0x7f045c14a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 >> 172.21.15.89:0/1580917010 conn(0x7f045c0d0bf0 msgr2=0x7f045c073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 shutdown_connections 2023-11-24T06:19:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:14.915+0000 7f0462f6d700 1 -- 172.21.15.89:0/1580917010 wait complete. 2023-11-24T06:19:14.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:15.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.050+0000 7fd02a32a700 1 Processor -- start 2023-11-24T06:19:15.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.050+0000 7fd02a32a700 1 -- start start 2023-11-24T06:19:15.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd02a32a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0240746a0 0x7fd0240b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd02a32a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240b3c60 con 0x7fd02415e160 2023-11-24T06:19:15.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd02a32a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240b58c0 con 0x7fd0240afc90 2023-11-24T06:19:15.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd02a32a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0240b4500 con 0x7fd0240746a0 2023-11-24T06:19:15.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd0237fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0240746a0 0x7fd0240b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd023fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd02415e160 0x7fd024073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39380/0 (socket says 172.21.15.89:39380) 2023-11-24T06:19:15.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd023fff700 1 -- 172.21.15.89:0/1313924362 learned_addr learned my addr 172.21.15.89:0/1313924362 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:15.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd0237fe700 1 -- 172.21.15.89:0/1313924362 >> v1:172.21.15.123:6789/0 conn(0x7fd0240afc90 legacy=0x7fd02406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd0237fe700 1 -- 172.21.15.89:0/1313924362 >> v1:172.21.15.89:6789/0 conn(0x7fd02415e160 legacy=0x7fd024073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd0237fe700 1 -- 172.21.15.89:0/1313924362 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02406bdc0 con 0x7fd0240746a0 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd022ffd700 1 -- 172.21.15.89:0/1313924362 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403192172 0 0) 0x7fd0240b3c60 con 0x7fd02415e160 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd0237fe700 1 --2- 172.21.15.89:0/1313924362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0240b5da0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd014009840 tx=0x7fd014009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4869af00831b2885 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.051+0000 7fd022ffd700 1 -- 172.21.15.89:0/1313924362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd014004230 con 0x7fd0240746a0 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd022ffd700 1 -- 172.21.15.89:0/1313924362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd014004390 con 0x7fd0240746a0 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd022ffd700 1 -- 172.21.15.89:0/1313924362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0140036a0 con 0x7fd0240746a0 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd02a32a700 1 -- 172.21.15.89:0/1313924362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 msgr2=0x7fd0240b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd02a32a700 1 --2- 172.21.15.89:0/1313924362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0240b5da0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd014009840 tx=0x7fd014009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd02a32a700 1 -- 172.21.15.89:0/1313924362 shutdown_connections 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd02a32a700 1 --2- 172.21.15.89:0/1313924362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0240b5da0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd02a32a700 1 -- 172.21.15.89:0/1313924362 >> 172.21.15.89:0/1313924362 conn(0x7fd0240d0bf0 msgr2=0x7fd0240d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:15.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd02a32a700 1 -- 172.21.15.89:0/1313924362 shutdown_connections 2023-11-24T06:19:15.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.052+0000 7fd02a32a700 1 -- 172.21.15.89:0/1313924362 wait complete. 2023-11-24T06:19:15.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.053+0000 7fd02a32a700 1 Processor -- start 2023-11-24T06:19:15.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.053+0000 7fd02a32a700 1 -- start start 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd02a32a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0241f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd02a32a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0240afc90 0x7fd024153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd02a32a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02415e160 0x7fd0241525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd02a32a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02415e750 con 0x7fd0240746a0 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd02a32a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02415e9a0 con 0x7fd02415e160 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd02a32a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02415ec80 con 0x7fd0240afc90 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd023fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0241f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd023fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0241f4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35882/0 (socket says 172.21.15.89:35882) 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd023fff700 1 -- 172.21.15.89:0/666662596 learned_addr learned my addr 172.21.15.89:0/666662596 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:15.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd0237fe700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0240afc90 0x7fd024153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.054+0000 7fd0288c7700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02415e160 0x7fd0241525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd023fff700 1 -- 172.21.15.89:0/666662596 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0240afc90 msgr2=0x7fd024153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd023fff700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0240afc90 0x7fd024153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd023fff700 1 -- 172.21.15.89:0/666662596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02415e160 msgr2=0x7fd0241525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd023fff700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02415e160 0x7fd0241525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd023fff700 1 -- 172.21.15.89:0/666662596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0140094a0 con 0x7fd0240746a0 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd023fff700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0241f4f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd01800c550 tx=0x7fd01800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd020ff9700 1 -- 172.21.15.89:0/666662596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018004c80 con 0x7fd0240746a0 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd020ff9700 1 -- 172.21.15.89:0/666662596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd018004e00 con 0x7fd0240746a0 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd020ff9700 1 -- 172.21.15.89:0/666662596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018007310 con 0x7fd0240746a0 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02415ef60 con 0x7fd0240746a0 2023-11-24T06:19:15.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.055+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02415f520 con 0x7fd0240746a0 2023-11-24T06:19:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.056+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0240b51c0 con 0x7fd0240746a0 2023-11-24T06:19:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.056+0000 7fd020ff9700 1 -- 172.21.15.89:0/666662596 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd018007470 con 0x7fd0240746a0 2023-11-24T06:19:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.057+0000 7fd020ff9700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd01006aed0 0x7fd01006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.057+0000 7fd020ff9700 1 -- 172.21.15.89:0/666662596 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0180892c0 con 0x7fd0240746a0 2023-11-24T06:19:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.057+0000 7fd0237fe700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd01006aed0 0x7fd01006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.057+0000 7fd0237fe700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd01006aed0 0x7fd01006d390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd014000c00 tx=0x7fd01401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.061+0000 7fd020ff9700 1 -- 172.21.15.89:0/666662596 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd018054370 con 0x7fd0240746a0 2023-11-24T06:19:15.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.252+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd02406f990 con 0x7fd0240746a0 2023-11-24T06:19:15.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.252+0000 7fd020ff9700 1 -- 172.21.15.89:0/666662596 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd018057990 con 0x7fd0240746a0 2023-11-24T06:19:15.255 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd01006aed0 msgr2=0x7fd01006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd01006aed0 0x7fd01006d390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd014000c00 tx=0x7fd01401c040 comp rx=0 tx=0).stop 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 msgr2=0x7fd0241f4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0241f4f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd01800c550 tx=0x7fd01800c910 comp rx=0 tx=0).stop 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 shutdown_connections 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd01006aed0 0x7fd01006d390 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02415e160 0x7fd0241525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0240746a0 0x7fd0241f4f90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 --2- 172.21.15.89:0/666662596 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0240afc90 0x7fd024153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 >> 172.21.15.89:0/666662596 conn(0x7fd0240d0bf0 msgr2=0x7fd0240d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:15.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 shutdown_connections 2023-11-24T06:19:15.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.254+0000 7fd02a32a700 1 -- 172.21.15.89:0/666662596 wait complete. 2023-11-24T06:19:15.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:15.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.389+0000 7f9340172700 1 Processor -- start 2023-11-24T06:19:15.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.389+0000 7f9340172700 1 -- start start 2023-11-24T06:19:15.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f9340172700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f933815be20 0x7f93380af790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f9340172700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93380b4e00 0x7f93380b5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f9340172700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f93380cd840 0x7f93380cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f9340172700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93380ce210 con 0x7f933815be20 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f9340172700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93380b4970 con 0x7f93380cd840 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f9340172700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93380cfed0 con 0x7f93380b4e00 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f933df0e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f933815be20 0x7f93380af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f933df0e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f933815be20 0x7f93380af790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35890/0 (socket says 172.21.15.89:35890) 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f933df0e700 1 -- 172.21.15.89:0/2370991798 learned_addr learned my addr 172.21.15.89:0/2370991798 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f933df0e700 1 -- 172.21.15.89:0/2370991798 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93380b4e00 msgr2=0x7f93380b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f933df0e700 1 --2- 172.21.15.89:0/2370991798 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93380b4e00 0x7f93380b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f933df0e700 1 -- 172.21.15.89:0/2370991798 >> v2:172.21.15.123:3300/0 conn(0x7f93380cd840 msgr2=0x7f93380cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f933df0e700 1 --2- 172.21.15.89:0/2370991798 >> v2:172.21.15.123:3300/0 conn(0x7f93380cd840 0x7f93380cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.390+0000 7f933df0e700 1 -- 172.21.15.89:0/2370991798 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933806f940 con 0x7f933815be20 2023-11-24T06:19:15.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f933df0e700 1 --2- 172.21.15.89:0/2370991798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f933815be20 0x7f93380af790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9338073750 tx=0x7f9328009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e9b893fd0576f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f933cf0c700 1 -- 172.21.15.89:0/2370991798 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800b870 con 0x7f933815be20 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f933cf0c700 1 -- 172.21.15.89:0/2370991798 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932800b9d0 con 0x7f933815be20 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f933cf0c700 1 -- 172.21.15.89:0/2370991798 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800bd60 con 0x7f933815be20 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 -- 172.21.15.89:0/2370991798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f933815be20 msgr2=0x7f93380af790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 --2- 172.21.15.89:0/2370991798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f933815be20 0x7f93380af790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9338073750 tx=0x7f9328009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 -- 172.21.15.89:0/2370991798 shutdown_connections 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 --2- 172.21.15.89:0/2370991798 >> v2:172.21.15.123:3300/0 conn(0x7f93380cd840 0x7f93380cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 --2- 172.21.15.89:0/2370991798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f933815be20 0x7f93380af790 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 --2- 172.21.15.89:0/2370991798 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93380b4e00 0x7f93380b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 -- 172.21.15.89:0/2370991798 >> 172.21.15.89:0/2370991798 conn(0x7f93380b2050 msgr2=0x7f93380b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 -- 172.21.15.89:0/2370991798 shutdown_connections 2023-11-24T06:19:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.391+0000 7f9340172700 1 -- 172.21.15.89:0/2370991798 wait complete. 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.392+0000 7f9340172700 1 Processor -- start 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f9340172700 1 -- start start 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f9340172700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380b4e00 0x7f93381f7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f9340172700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93380cd840 0x7f93381f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f9340172700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f933815be20 0x7f933806e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f9340172700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93381f85d0 con 0x7f93380cd840 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f9340172700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93381f40d0 con 0x7f93380b4e00 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f9340172700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93381f8010 con 0x7f933815be20 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f933d70d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93380cd840 0x7f93381f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f933d70d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93380cd840 0x7f93381f4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35896/0 (socket says 172.21.15.89:35896) 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f933d70d700 1 -- 172.21.15.89:0/1870665776 learned_addr learned my addr 172.21.15.89:0/1870665776 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f933df0e700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380b4e00 0x7f93381f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.393+0000 7f933e70f700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f933815be20 0x7f933806e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f933d70d700 1 -- 172.21.15.89:0/1870665776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f933815be20 msgr2=0x7f933806e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f933d70d700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f933815be20 0x7f933806e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f933d70d700 1 -- 172.21.15.89:0/1870665776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380b4e00 msgr2=0x7f93381f7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f933d70d700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380b4e00 0x7f93381f7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f933d70d700 1 -- 172.21.15.89:0/1870665776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9328009550 con 0x7f93380cd840 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f933d70d700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93380cd840 0x7f93381f4b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f933400b4a0 tx=0x7f933400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f932e7fc700 1 -- 172.21.15.89:0/1870665776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334011830 con 0x7f93380cd840 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f932e7fc700 1 -- 172.21.15.89:0/1870665776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334011990 con 0x7f93380cd840 2023-11-24T06:19:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93381f82f0 con 0x7f93380cd840 2023-11-24T06:19:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f932e7fc700 1 -- 172.21.15.89:0/1870665776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334011d40 con 0x7f93380cd840 2023-11-24T06:19:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.394+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93381f45f0 con 0x7f93380cd840 2023-11-24T06:19:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.395+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93381f47b0 con 0x7f93380cd840 2023-11-24T06:19:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.395+0000 7f932e7fc700 1 -- 172.21.15.89:0/1870665776 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f933401a400 con 0x7f93380cd840 2023-11-24T06:19:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.396+0000 7f932e7fc700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f931c06aed0 0x7f931c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.396+0000 7f932e7fc700 1 -- 172.21.15.89:0/1870665776 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93340895d0 con 0x7f93380cd840 2023-11-24T06:19:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.396+0000 7f933df0e700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f931c06aed0 0x7f931c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.396+0000 7f933df0e700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f931c06aed0 0x7f931c06d390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9328000c00 tx=0x7f9328008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.401+0000 7f932e7fc700 1 -- 172.21.15.89:0/1870665776 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9334054690 con 0x7f93380cd840 2023-11-24T06:19:15.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.593+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9338068d90 con 0x7f93380cd840 2023-11-24T06:19:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.594+0000 7f932e7fc700 1 -- 172.21.15.89:0/1870665776 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9334057cb0 con 0x7f93380cd840 2023-11-24T06:19:15.597 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:15.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f931c06aed0 msgr2=0x7f931c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f931c06aed0 0x7f931c06d390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9328000c00 tx=0x7f9328008030 comp rx=0 tx=0).stop 2023-11-24T06:19:15.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93380cd840 msgr2=0x7f93381f4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93380cd840 0x7f93381f4b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f933400b4a0 tx=0x7f933400b860 comp rx=0 tx=0).stop 2023-11-24T06:19:15.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 shutdown_connections 2023-11-24T06:19:15.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f933815be20 0x7f933806e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f931c06aed0 0x7f931c06d390 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380b4e00 0x7f93381f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 --2- 172.21.15.89:0/1870665776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93380cd840 0x7f93381f4b10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 >> 172.21.15.89:0/1870665776 conn(0x7f93380b2050 msgr2=0x7f933806d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:15.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 shutdown_connections 2023-11-24T06:19:15.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.596+0000 7f9340172700 1 -- 172.21.15.89:0/1870665776 wait complete. 2023-11-24T06:19:15.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:15.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.729+0000 7fa65ecdc700 1 Processor -- start 2023-11-24T06:19:15.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.729+0000 7fa65ecdc700 1 -- start start 2023-11-24T06:19:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65ecdc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa65815e560 0x7fa65806d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65ecdc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6580cc5c0 con 0x7fa6580b49b0 2023-11-24T06:19:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65ecdc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6580717b0 con 0x7fa658151500 2023-11-24T06:19:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65ecdc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa658071d70 con 0x7fa65815e560 2023-11-24T06:19:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65d279700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa65815e560 0x7fa65806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa657fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa6580b49b0 0x7fa65815de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39410/0 (socket says 172.21.15.89:39410) 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa657fff700 1 -- 172.21.15.89:0/486766168 learned_addr learned my addr 172.21.15.89:0/486766168 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65d279700 1 -- 172.21.15.89:0/486766168 >> v1:172.21.15.123:6789/0 conn(0x7fa658151500 legacy=0x7fa6580cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65d279700 1 -- 172.21.15.89:0/486766168 >> v1:172.21.15.89:6789/0 conn(0x7fa6580b49b0 legacy=0x7fa65815de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa6577fe700 1 -- 172.21.15.89:0/486766168 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944856388 0 0) 0x7fa6580cc5c0 con 0x7fa6580b49b0 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65d279700 1 -- 172.21.15.89:0/486766168 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa658071a90 con 0x7fa65815e560 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.730+0000 7fa65d279700 1 --2- 172.21.15.89:0/486766168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa65815e560 0x7fa65806d350 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa65000aed0 tx=0x7fa65000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b627cb1b0493a53 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa6577fe700 1 -- 172.21.15.89:0/486766168 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65000f070 con 0x7fa65815e560 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa6577fe700 1 -- 172.21.15.89:0/486766168 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65000a2e0 con 0x7fa65815e560 2023-11-24T06:19:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa6577fe700 1 -- 172.21.15.89:0/486766168 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650003c60 con 0x7fa65815e560 2023-11-24T06:19:15.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa65ecdc700 1 -- 172.21.15.89:0/486766168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa65815e560 msgr2=0x7fa65806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa65ecdc700 1 --2- 172.21.15.89:0/486766168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa65815e560 0x7fa65806d350 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa65000aed0 tx=0x7fa65000c330 comp rx=0 tx=0).stop 2023-11-24T06:19:15.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa65ecdc700 1 -- 172.21.15.89:0/486766168 shutdown_connections 2023-11-24T06:19:15.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa65ecdc700 1 --2- 172.21.15.89:0/486766168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa65815e560 0x7fa65806d350 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa65ecdc700 1 -- 172.21.15.89:0/486766168 >> 172.21.15.89:0/486766168 conn(0x7fa6580b1740 msgr2=0x7fa6580b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:15.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa65ecdc700 1 -- 172.21.15.89:0/486766168 shutdown_connections 2023-11-24T06:19:15.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.731+0000 7fa65ecdc700 1 -- 172.21.15.89:0/486766168 wait complete. 2023-11-24T06:19:15.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.732+0000 7fa65ecdc700 1 Processor -- start 2023-11-24T06:19:15.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.732+0000 7fa65ecdc700 1 -- start start 2023-11-24T06:19:15.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ecdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6580b49b0 0x7fa658151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ecdc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa658151500 0x7fa658157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ecdc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa65815e560 0x7fa6580afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ecdc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65813dd40 con 0x7fa6580b49b0 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ecdc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa658151e80 con 0x7fa658151500 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ecdc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65813d130 con 0x7fa65815e560 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ca78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6580b49b0 0x7fa658151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ca78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6580b49b0 0x7fa658151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35910/0 (socket says 172.21.15.89:35910) 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.733+0000 7fa65ca78700 1 -- 172.21.15.89:0/847105901 learned_addr learned my addr 172.21.15.89:0/847105901 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:15.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa657fff700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa658151500 0x7fa658157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65d279700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa65815e560 0x7fa6580afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65ca78700 1 -- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa65815e560 msgr2=0x7fa6580afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65ca78700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa65815e560 0x7fa6580afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65ca78700 1 -- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa658151500 msgr2=0x7fa658157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65ca78700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa658151500 0x7fa658157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65ca78700 1 -- 172.21.15.89:0/847105901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65000ab80 con 0x7fa6580b49b0 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65d279700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa65815e560 0x7fa6580afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa657fff700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa658151500 0x7fa658157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65ca78700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6580b49b0 0x7fa658151940 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c500 tx=0x7fa64c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa6557fa700 1 -- 172.21.15.89:0/847105901 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c011790 con 0x7fa6580b49b0 2023-11-24T06:19:15.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa6557fa700 1 -- 172.21.15.89:0/847105901 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c0118f0 con 0x7fa6580b49b0 2023-11-24T06:19:15.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65813d410 con 0x7fa6580b49b0 2023-11-24T06:19:15.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa6557fa700 1 -- 172.21.15.89:0/847105901 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c011ca0 con 0x7fa6580b49b0 2023-11-24T06:19:15.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.734+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65813d9d0 con 0x7fa6580b49b0 2023-11-24T06:19:15.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.735+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa658158180 con 0x7fa6580b49b0 2023-11-24T06:19:15.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.735+0000 7fa6557fa700 1 -- 172.21.15.89:0/847105901 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa64c01c460 con 0x7fa6580b49b0 2023-11-24T06:19:15.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.736+0000 7fa6557fa700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa63c06aed0 0x7fa63c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:15.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.736+0000 7fa6557fa700 1 -- 172.21.15.89:0/847105901 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa64c0894b0 con 0x7fa6580b49b0 2023-11-24T06:19:15.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.736+0000 7fa657fff700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa63c06aed0 0x7fa63c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:15.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.736+0000 7fa657fff700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa63c06aed0 0x7fa63c06d390 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa6580b09a0 tx=0x7fa644009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.740+0000 7fa6557fa700 1 -- 172.21.15.89:0/847105901 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa64c0544c0 con 0x7fa6580b49b0 2023-11-24T06:19:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.932+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6580751e0 con 0x7fa6580b49b0 2023-11-24T06:19:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.933+0000 7fa6557fa700 1 -- 172.21.15.89:0/847105901 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7fa64c057ae0 con 0x7fa6580b49b0 2023-11-24T06:19:15.936 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:15.937 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.935+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa63c06aed0 msgr2=0x7fa63c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.935+0000 7fa65ecdc700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa63c06aed0 0x7fa63c06d390 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa6580b09a0 tx=0x7fa644009040 comp rx=0 tx=0).stop 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.935+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6580b49b0 msgr2=0x7fa658151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.935+0000 7fa65ecdc700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6580b49b0 0x7fa658151940 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c500 tx=0x7fa64c00c8c0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.935+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 shutdown_connections 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.935+0000 7fa65ecdc700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa65815e560 0x7fa6580afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.935+0000 7fa65ecdc700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6580b49b0 0x7fa658151940 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.935+0000 7fa65ecdc700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa63c06aed0 0x7fa63c06d390 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.936+0000 7fa65ecdc700 1 --2- 172.21.15.89:0/847105901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa658151500 0x7fa658157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.936+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 >> 172.21.15.89:0/847105901 conn(0x7fa6580b1740 msgr2=0x7fa65814e680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.936+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 shutdown_connections 2023-11-24T06:19:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:15.936+0000 7fa65ecdc700 1 -- 172.21.15.89:0/847105901 wait complete. 2023-11-24T06:19:15.951 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:15.951 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:16.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.070+0000 7f22f8436700 1 Processor -- start 2023-11-24T06:19:16.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.070+0000 7f22f8436700 1 -- start start 2023-11-24T06:19:16.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.071+0000 7f22f8436700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.071+0000 7f22f8436700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f22f00b3c60 0x7f22f00b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.071+0000 7f22f8436700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f00ae5e0 con 0x7f22f01587f0 2023-11-24T06:19:16.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.071+0000 7f22f8436700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f00b4610 con 0x7f22f00b3c60 2023-11-24T06:19:16.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.071+0000 7f22f8436700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f015be30 con 0x7f22f015b370 2023-11-24T06:19:16.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.071+0000 7f22f69d3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f22f00b3c60 0x7f22f00b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f69d3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f22f00b3c60 0x7f22f00b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35922/0 (socket says 172.21.15.89:35922) 2023-11-24T06:19:16.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f69d3700 1 -- 172.21.15.89:0/3654182507 learned_addr learned my addr 172.21.15.89:0/3654182507 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f59d1700 1 --2- 172.21.15.89:0/3654182507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f69d3700 1 -- 172.21.15.89:0/3654182507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 msgr2=0x7f22f015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f69d3700 1 --2- 172.21.15.89:0/3654182507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f015b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f69d3700 1 -- 172.21.15.89:0/3654182507 >> v1:172.21.15.123:6789/0 conn(0x7f22f01587f0 legacy=0x7f22f006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f69d3700 1 -- 172.21.15.89:0/3654182507 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f015c110 con 0x7f22f00b3c60 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f59d1700 1 --2- 172.21.15.89:0/3654182507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f69d3700 1 --2- 172.21.15.89:0/3654182507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f00b40d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d150 tx=0x7f22e800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcaee3dcc92b67bd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f51d0700 1 -- 172.21.15.89:0/3654182507 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8010070 con 0x7f22f00b3c60 2023-11-24T06:19:16.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f51d0700 1 -- 172.21.15.89:0/3654182507 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e800ddc0 con 0x7f22f00b3c60 2023-11-24T06:19:16.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.072+0000 7f22f51d0700 1 -- 172.21.15.89:0/3654182507 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800b310 con 0x7f22f00b3c60 2023-11-24T06:19:16.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.073+0000 7f22f8436700 1 -- 172.21.15.89:0/3654182507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 msgr2=0x7f22f00b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.073+0000 7f22f8436700 1 --2- 172.21.15.89:0/3654182507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f00b40d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d150 tx=0x7f22e800d460 comp rx=0 tx=0).stop 2023-11-24T06:19:16.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.073+0000 7f22f8436700 1 -- 172.21.15.89:0/3654182507 shutdown_connections 2023-11-24T06:19:16.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.073+0000 7f22f8436700 1 --2- 172.21.15.89:0/3654182507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f00b40d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.073+0000 7f22f8436700 1 --2- 172.21.15.89:0/3654182507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.073+0000 7f22f8436700 1 -- 172.21.15.89:0/3654182507 >> 172.21.15.89:0/3654182507 conn(0x7f22f00d0bf0 msgr2=0x7f22f00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:16.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.073+0000 7f22f8436700 1 -- 172.21.15.89:0/3654182507 shutdown_connections 2023-11-24T06:19:16.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.073+0000 7f22f8436700 1 -- 172.21.15.89:0/3654182507 wait complete. 2023-11-24T06:19:16.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.074+0000 7f22f8436700 1 Processor -- start 2023-11-24T06:19:16.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.074+0000 7f22f8436700 1 -- start start 2023-11-24T06:19:16.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f8436700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f0152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f8436700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f01587f0 0x7f22f01447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f8436700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f00b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f8436700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0144d10 con 0x7f22f00b3c60 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f8436700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f00b0e20 con 0x7f22f01587f0 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f8436700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0152fd0 con 0x7f22f015b370 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f0152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f0152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35924/0 (socket says 172.21.15.89:35924) 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 -- 172.21.15.89:0/535863148 learned_addr learned my addr 172.21.15.89:0/535863148 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f69d3700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f00b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f59d1700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f01587f0 0x7f22f01447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 -- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 msgr2=0x7f22f00b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f00b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 -- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f01587f0 msgr2=0x7f22f01447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f01587f0 0x7f22f01447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 -- 172.21.15.89:0/535863148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e800bd40 con 0x7f22f00b3c60 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f59d1700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f01587f0 0x7f22f01447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f69d3700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f00b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f61d2700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f0152910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f22e000c4e0 tx=0x7f22e000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22e6ffd700 1 -- 172.21.15.89:0/535863148 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e00052d0 con 0x7f22f00b3c60 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22e6ffd700 1 -- 172.21.15.89:0/535863148 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e0005430 con 0x7f22f00b3c60 2023-11-24T06:19:16.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f01532b0 con 0x7f22f00b3c60 2023-11-24T06:19:16.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22e6ffd700 1 -- 172.21.15.89:0/535863148 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e00038c0 con 0x7f22f00b3c60 2023-11-24T06:19:16.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.075+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f0153870 con 0x7f22f00b3c60 2023-11-24T06:19:16.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.076+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f0068fb0 con 0x7f22f00b3c60 2023-11-24T06:19:16.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.077+0000 7f22e6ffd700 1 -- 172.21.15.89:0/535863148 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f22e0003a20 con 0x7f22f00b3c60 2023-11-24T06:19:16.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.077+0000 7f22e6ffd700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22d406aed0 0x7f22d406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.077+0000 7f22e6ffd700 1 -- 172.21.15.89:0/535863148 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22e0088ab0 con 0x7f22f00b3c60 2023-11-24T06:19:16.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.077+0000 7f22f59d1700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22d406aed0 0x7f22d406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.078+0000 7f22f59d1700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22d406aed0 0x7f22d406d390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f22f014e7c0 tx=0x7f22ec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.081+0000 7f22e6ffd700 1 -- 172.21.15.89:0/535863148 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22e0053b60 con 0x7f22f00b3c60 2023-11-24T06:19:16.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.271+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22f00d1490 con 0x7f22f00b3c60 2023-11-24T06:19:16.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.271+0000 7f22e6ffd700 1 -- 172.21.15.89:0/535863148 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22e00170a0 con 0x7f22f00b3c60 2023-11-24T06:19:16.274 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:16.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22d406aed0 msgr2=0x7f22d406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22d406aed0 0x7f22d406d390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f22f014e7c0 tx=0x7f22ec008040 comp rx=0 tx=0).stop 2023-11-24T06:19:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 msgr2=0x7f22f0152910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f0152910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f22e000c4e0 tx=0x7f22e000c8a0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 shutdown_connections 2023-11-24T06:19:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f015b370 0x7f22f00b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f00b3c60 0x7f22f0152910 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22d406aed0 0x7f22d406d390 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 --2- 172.21.15.89:0/535863148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f01587f0 0x7f22f01447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 >> 172.21.15.89:0/535863148 conn(0x7f22f00d0bf0 msgr2=0x7f22f0156910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 shutdown_connections 2023-11-24T06:19:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.273+0000 7f22f8436700 1 -- 172.21.15.89:0/535863148 wait complete. 2023-11-24T06:19:16.288 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:16.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.408+0000 7fe76e9f0700 1 Processor -- start 2023-11-24T06:19:16.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.408+0000 7fe76e9f0700 1 -- start start 2023-11-24T06:19:16.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe76e9f0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5da0 0x7fe7680afc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe76e9f0700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680b3c60 con 0x7fe7680746a0 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe76e9f0700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680b58c0 con 0x7fe76815e160 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe76e9f0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680b4500 con 0x7fe7680b5da0 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe7677fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe7680746a0 0x7fe76806d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39438/0 (socket says 172.21.15.89:39438) 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe7677fe700 1 -- 172.21.15.89:0/3792392956 learned_addr learned my addr 172.21.15.89:0/3792392956 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925719342 0 0) 0x7fe7680b3c60 con 0x7fe7680746a0 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7500035b0 con 0x7fe7680746a0 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe76cf8d700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5da0 0x7fe7680afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1707267229 0 0) 0x7fe7500035b0 con 0x7fe7680746a0 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5da0 msgr2=0x7fe7680afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe766ffd700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5da0 0x7fe7680afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 >> v1:172.21.15.123:6789/0 conn(0x7fe76815e160 legacy=0x7fe768073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76806bdc0 con 0x7fe7680746a0 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.409+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fe758001670 con 0x7fe7680746a0 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 >> v1:172.21.15.89:6789/0 conn(0x7fe7680746a0 legacy=0x7fe76806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a30 0x7fe750003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5da0 0x7fe750005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680b4500 con 0x7fe750003a30 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75000bac0 con 0x7fe750004850 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75000bc80 con 0x7fe7680b5da0 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe767fff700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a30 0x7fe750003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe76cf8d700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5da0 0x7fe750005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe7677fe700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe767fff700 1 -- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5da0 msgr2=0x7fe750005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe767fff700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5da0 0x7fe750005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe767fff700 1 -- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750004850 msgr2=0x7fe750004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe767fff700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe767fff700 1 -- 172.21.15.89:0/3792392956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500035b0 con 0x7fe750003a30 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe767fff700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a30 0x7fe750003e50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe75c007fd0 tx=0x7fe750005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593790c2b925037b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0012f0 con 0x7fe750003a30 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c0052f0 con 0x7fe750003a30 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.410+0000 7fe766ffd700 1 -- 172.21.15.89:0/3792392956 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c006860 con 0x7fe750003a30 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.411+0000 7fe76e9f0700 1 -- 172.21.15.89:0/3792392956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a30 msgr2=0x7fe750003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.411+0000 7fe76e9f0700 1 --2- 172.21.15.89:0/3792392956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a30 0x7fe750003e50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe75c007fd0 tx=0x7fe750005880 comp rx=0 tx=0).stop 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.411+0000 7fe76cf8d700 1 -- 172.21.15.89:0/3792392956 reap_dead start 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.411+0000 7fe76e9f0700 1 -- 172.21.15.89:0/3792392956 shutdown_connections 2023-11-24T06:19:16.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.411+0000 7fe76e9f0700 1 -- 172.21.15.89:0/3792392956 >> 172.21.15.89:0/3792392956 conn(0x7fe7680d0bf0 msgr2=0x7fe7680d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:16.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.411+0000 7fe76e9f0700 1 -- 172.21.15.89:0/3792392956 shutdown_connections 2023-11-24T06:19:16.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.411+0000 7fe76e9f0700 1 -- 172.21.15.89:0/3792392956 wait complete. 2023-11-24T06:19:16.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.412+0000 7fe76e9f0700 1 Processor -- start 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.412+0000 7fe76e9f0700 1 -- start start 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe76e9f0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a30 0x7fe76814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe76e9f0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7680c5bd0 0x7fe7680c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe76e9f0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe76814e2a0 0x7fe7680c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe76e9f0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680c6d50 con 0x7fe7680c5bd0 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe76e9f0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680c6140 con 0x7fe750003a30 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe76e9f0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680c6420 con 0x7fe76814e2a0 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7680c5bd0 0x7fe7680c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7680c5bd0 0x7fe7680c71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35946/0 (socket says 172.21.15.89:35946) 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 -- 172.21.15.89:0/208082415 learned_addr learned my addr 172.21.15.89:0/208082415 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe767fff700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a30 0x7fe76814dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe76cf8d700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe76814e2a0 0x7fe7680c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 -- 172.21.15.89:0/208082415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe76814e2a0 msgr2=0x7fe7680c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe76814e2a0 0x7fe7680c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 -- 172.21.15.89:0/208082415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a30 msgr2=0x7fe76814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a30 0x7fe76814dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 -- 172.21.15.89:0/208082415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680c6700 con 0x7fe7680c5bd0 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.413+0000 7fe7677fe700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7680c5bd0 0x7fe7680c71d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe7580072c0 tx=0x7fe758007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.414+0000 7fe764ff9700 1 -- 172.21.15.89:0/208082415 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580077a0 con 0x7fe7680c5bd0 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.414+0000 7fe764ff9700 1 -- 172.21.15.89:0/208082415 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75800f850 con 0x7fe7680c5bd0 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.414+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7680c69e0 con 0x7fe7680c5bd0 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.414+0000 7fe764ff9700 1 -- 172.21.15.89:0/208082415 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75800fc00 con 0x7fe7680c5bd0 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.414+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76814e9b0 con 0x7fe7680c5bd0 2023-11-24T06:19:16.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.415+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe768068fb0 con 0x7fe7680c5bd0 2023-11-24T06:19:16.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.415+0000 7fe764ff9700 1 -- 172.21.15.89:0/208082415 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe75800fd60 con 0x7fe7680c5bd0 2023-11-24T06:19:16.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.415+0000 7fe764ff9700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.416+0000 7fe767fff700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.416+0000 7fe764ff9700 1 -- 172.21.15.89:0/208082415 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe758089580 con 0x7fe7680c5bd0 2023-11-24T06:19:16.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.416+0000 7fe767fff700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe75c014040 tx=0x7fe75c011000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.420+0000 7fe764ff9700 1 -- 172.21.15.89:0/208082415 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe758054630 con 0x7fe7680c5bd0 2023-11-24T06:19:16.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.608+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe7680b4970 con 0x7fe7680c5bd0 2023-11-24T06:19:16.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.609+0000 7fe764ff9700 1 -- 172.21.15.89:0/208082415 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe758057c50 con 0x7fe7680c5bd0 2023-11-24T06:19:16.612 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 msgr2=0x7fe75406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe75c014040 tx=0x7fe75c011000 comp rx=0 tx=0).stop 2023-11-24T06:19:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7680c5bd0 msgr2=0x7fe7680c71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7680c5bd0 0x7fe7680c71d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe7580072c0 tx=0x7fe758007680 comp rx=0 tx=0).stop 2023-11-24T06:19:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 shutdown_connections 2023-11-24T06:19:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe76814e2a0 0x7fe7680c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a30 0x7fe76814dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 --2- 172.21.15.89:0/208082415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7680c5bd0 0x7fe7680c71d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 >> 172.21.15.89:0/208082415 conn(0x7fe7680d0bf0 msgr2=0x7fe7680afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 shutdown_connections 2023-11-24T06:19:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.611+0000 7fe76e9f0700 1 -- 172.21.15.89:0/208082415 wait complete. 2023-11-24T06:19:16.626 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.746+0000 7f1875a13700 1 Processor -- start 2023-11-24T06:19:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.746+0000 7f1875a13700 1 -- start start 2023-11-24T06:19:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.747+0000 7f1875a13700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f18700d45d0 0x7f18700b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.747+0000 7f1875a13700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700d2930 con 0x7f18700d39d0 2023-11-24T06:19:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.747+0000 7f1875a13700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700d2c10 con 0x7f187015e140 2023-11-24T06:19:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.747+0000 7f1875a13700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187015e520 con 0x7f18700d45d0 2023-11-24T06:19:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.747+0000 7f186f7fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f18700d39d0 0x7f18700b7610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39460/0 (socket says 172.21.15.89:39460) 2023-11-24T06:19:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.747+0000 7f186f7fe700 1 -- 172.21.15.89:0/2863066955 learned_addr learned my addr 172.21.15.89:0/2863066955 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918976273 0 0) 0x7f18700d2930 con 0x7f18700d39d0 2023-11-24T06:19:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18580035b0 con 0x7f18700d39d0 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186e7fc700 1 --2- 172.21.15.89:0/2863066955 >> v2:172.21.15.123:3300/0 conn(0x7f18700d45d0 0x7f18700b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3459557262 0 0) 0x7f18580035b0 con 0x7f18700d39d0 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 >> v2:172.21.15.123:3300/0 conn(0x7f18700d45d0 msgr2=0x7f18700b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 --2- 172.21.15.89:0/2863066955 >> v2:172.21.15.123:3300/0 conn(0x7f18700d45d0 0x7f18700b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 >> v1:172.21.15.123:6789/0 conn(0x7f187015e140 legacy=0x7f18700d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18700d2ef0 con 0x7f18700d39d0 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f18680019f0 con 0x7f18700d39d0 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 >> v1:172.21.15.89:6789/0 conn(0x7f18700d39d0 legacy=0x7f18700b7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f1858003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1858004850 0x7f1858004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1858005210 0x7f185800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187015e520 con 0x7f1858003a00 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185800c160 con 0x7f1858004850 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185800c320 con 0x7f1858005210 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186effd700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f1858003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186e7fc700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1858004850 0x7f1858004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186f7fe700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1858005210 0x7f185800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186effd700 1 -- 172.21.15.89:0/2863066955 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1858005210 msgr2=0x7f185800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186effd700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1858005210 0x7f185800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186effd700 1 -- 172.21.15.89:0/2863066955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1858004850 msgr2=0x7f1858004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186effd700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1858004850 0x7f1858004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186f7fe700 1 -- 172.21.15.89:0/2863066955 reap_dead start 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186effd700 1 -- 172.21.15.89:0/2863066955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18580035b0 con 0x7f1858003a00 2023-11-24T06:19:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.748+0000 7f186effd700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f1858003e20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f18600017c0 tx=0x7f1858005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb6f4b1401dea9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18600052d0 con 0x7f1858003a00 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1860005430 con 0x7f1858003a00 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f186dffb700 1 -- 172.21.15.89:0/2863066955 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18600038c0 con 0x7f1858003a00 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f1875a13700 1 -- 172.21.15.89:0/2863066955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 msgr2=0x7f1858003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f1875a13700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f1858003e20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f18600017c0 tx=0x7f1858005880 comp rx=0 tx=0).stop 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f1875a13700 1 -- 172.21.15.89:0/2863066955 shutdown_connections 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f1875a13700 1 --2- 172.21.15.89:0/2863066955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f1858003e20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f1875a13700 1 -- 172.21.15.89:0/2863066955 >> 172.21.15.89:0/2863066955 conn(0x7f187006b410 msgr2=0x7f187006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f1875a13700 1 -- 172.21.15.89:0/2863066955 shutdown_connections 2023-11-24T06:19:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.749+0000 7f1875a13700 1 -- 172.21.15.89:0/2863066955 wait complete. 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.751+0000 7f1875a13700 1 Processor -- start 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.751+0000 7f1875a13700 1 -- start start 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f1875a13700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f187015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f1875a13700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f18700ce370 0x7f18700cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f1875a13700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1870160760 0x7f1870160be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f1875a13700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1870161120 con 0x7f1858003a00 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f1875a13700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700cd4c0 con 0x7f1870160760 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f1875a13700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18701f3f20 con 0x7f18700ce370 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f187015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f187015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35964/0 (socket says 172.21.15.89:35964) 2023-11-24T06:19:16.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 -- 172.21.15.89:0/3622953422 learned_addr learned my addr 172.21.15.89:0/3622953422 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186e7fc700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f18700ce370 0x7f18700cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 -- 172.21.15.89:0/3622953422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f18700ce370 msgr2=0x7f18700cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f18700ce370 0x7f18700cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 -- 172.21.15.89:0/3622953422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1870160760 msgr2=0x7f1870160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1870160760 0x7f1870160be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 -- 172.21.15.89:0/3622953422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187015cab0 con 0x7f1858003a00 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.752+0000 7f186effd700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f187015c570 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1860001db0 tx=0x7f1860001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.753+0000 7f1857fff700 1 -- 172.21.15.89:0/3622953422 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1860001290 con 0x7f1858003a00 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.753+0000 7f1857fff700 1 -- 172.21.15.89:0/3622953422 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1860007040 con 0x7f1858003a00 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.753+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18700ccc20 con 0x7f1858003a00 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.753+0000 7f1857fff700 1 -- 172.21.15.89:0/3622953422 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18600073d0 con 0x7f1858003a00 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.753+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18700cd1e0 con 0x7f1858003a00 2023-11-24T06:19:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.754+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1870068fb0 con 0x7f1858003a00 2023-11-24T06:19:16.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.754+0000 7f1857fff700 1 -- 172.21.15.89:0/3622953422 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1860007530 con 0x7f1858003a00 2023-11-24T06:19:16.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.754+0000 7f1857fff700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f185c06aed0 0x7f185c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:16.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.755+0000 7f1857fff700 1 -- 172.21.15.89:0/3622953422 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1860089e20 con 0x7f1858003a00 2023-11-24T06:19:16.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.755+0000 7f186e7fc700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f185c06aed0 0x7f185c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:16.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.755+0000 7f186e7fc700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f185c06aed0 0x7f185c06d390 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1870161300 tx=0x7f1864008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:16.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.759+0000 7f1857fff700 1 -- 172.21.15.89:0/3622953422 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1860054f60 con 0x7f1858003a00 2023-11-24T06:19:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.948+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18701f3cf0 con 0x7f1858003a00 2023-11-24T06:19:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.948+0000 7f1857fff700 1 -- 172.21.15.89:0/3622953422 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18600170a0 con 0x7f1858003a00 2023-11-24T06:19:16.951 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f185c06aed0 msgr2=0x7f185c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f185c06aed0 0x7f185c06d390 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1870161300 tx=0x7f1864008040 comp rx=0 tx=0).stop 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 msgr2=0x7f187015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f187015c570 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1860001db0 tx=0x7f1860001120 comp rx=0 tx=0).stop 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 shutdown_connections 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f185c06aed0 0x7f185c06d390 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1870160760 0x7f1870160be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1858003a00 0x7f187015c570 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 --2- 172.21.15.89:0/3622953422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f18700ce370 0x7f18700cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.950+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 >> 172.21.15.89:0/3622953422 conn(0x7f187006b410 msgr2=0x7f18700d3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:16.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.951+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 shutdown_connections 2023-11-24T06:19:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:16.951+0000 7f1875a13700 1 -- 172.21.15.89:0/3622953422 wait complete. 2023-11-24T06:19:16.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.083+0000 7fdde1d13700 1 Processor -- start 2023-11-24T06:19:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.083+0000 7fdde1d13700 1 -- start start 2023-11-24T06:19:17.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdde1d13700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 0x7fdddc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdde1d13700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdddc0af4d0 0x7fdddc0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdde1d13700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0ae430 con 0x7fdddc15e080 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdde1d13700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc0af0b0 con 0x7fdddc0af4d0 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdde1d13700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0b05a0 con 0x7fdddc06c710 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdddbfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdddc0af4d0 0x7fdddc0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdddbfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdddc0af4d0 0x7fdddc0afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35976/0 (socket says 172.21.15.89:35976) 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdddbfff700 1 -- 172.21.15.89:0/3553454200 learned_addr learned my addr 172.21.15.89:0/3553454200 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fddd2ffd700 1 --2- 172.21.15.89:0/3553454200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 0x7fdddc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdddbfff700 1 -- 172.21.15.89:0/3553454200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 msgr2=0x7fdddc0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdddbfff700 1 --2- 172.21.15.89:0/3553454200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 0x7fdddc0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdddbfff700 1 -- 172.21.15.89:0/3553454200 >> v1:172.21.15.123:6789/0 conn(0x7fdddc15e080 legacy=0x7fdddc06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdddbfff700 1 -- 172.21.15.89:0/3553454200 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0ae710 con 0x7fdddc0af4d0 2023-11-24T06:19:17.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.084+0000 7fdddbfff700 1 --2- 172.21.15.89:0/3553454200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc0afb90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fddd4009e80 tx=0x7fddd400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9c8e06cbf05bb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdddaffd700 1 -- 172.21.15.89:0/3553454200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd400e070 con 0x7fdddc0af4d0 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdddaffd700 1 -- 172.21.15.89:0/3553454200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd400b980 con 0x7fdddc0af4d0 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdddaffd700 1 -- 172.21.15.89:0/3553454200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd400bd30 con 0x7fdddc0af4d0 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdde1d13700 1 -- 172.21.15.89:0/3553454200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 msgr2=0x7fdddc0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdde1d13700 1 --2- 172.21.15.89:0/3553454200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc0afb90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fddd4009e80 tx=0x7fddd400b330 comp rx=0 tx=0).stop 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdde1d13700 1 -- 172.21.15.89:0/3553454200 shutdown_connections 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdde1d13700 1 --2- 172.21.15.89:0/3553454200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc0afb90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdde1d13700 1 --2- 172.21.15.89:0/3553454200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 0x7fdddc0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdde1d13700 1 -- 172.21.15.89:0/3553454200 >> 172.21.15.89:0/3553454200 conn(0x7fdddc071b90 msgr2=0x7fdddc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdde1d13700 1 -- 172.21.15.89:0/3553454200 shutdown_connections 2023-11-24T06:19:17.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.085+0000 7fdde1d13700 1 -- 172.21.15.89:0/3553454200 wait complete. 2023-11-24T06:19:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.086+0000 7fdde1d13700 1 Processor -- start 2023-11-24T06:19:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.087+0000 7fdde1d13700 1 -- start start 2023-11-24T06:19:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.087+0000 7fdde1d13700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 0x7fdddc14da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.087+0000 7fdde1d13700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc1f7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.087+0000 7fdde1d13700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdddc15e080 0x7fdddc0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.087+0000 7fdde1d13700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0d02e0 con 0x7fdddc0af4d0 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.087+0000 7fdde1d13700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cfb00 con 0x7fdddc15e080 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.087+0000 7fdde1d13700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc157ed0 con 0x7fdddc06c710 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc1f7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc1f7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35982/0 (socket says 172.21.15.89:35982) 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 -- 172.21.15.89:0/2925332748 learned_addr learned my addr 172.21.15.89:0/2925332748 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fdddb7fe700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 0x7fdddc14da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fdddbfff700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdddc15e080 0x7fdddc0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 -- 172.21.15.89:0/2925332748 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 msgr2=0x7fdddc14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 0x7fdddc14da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 -- 172.21.15.89:0/2925332748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdddc15e080 msgr2=0x7fdddc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdddc15e080 0x7fdddc0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 -- 172.21.15.89:0/2925332748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd4009b30 con 0x7fdddc0af4d0 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd2ffd700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc1f7880 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fddc400e840 tx=0x7fddc400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd8ff9700 1 -- 172.21.15.89:0/2925332748 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4013610 con 0x7fdddc0af4d0 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd8ff9700 1 -- 172.21.15.89:0/2925332748 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc4013770 con 0x7fdddc0af4d0 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc14df50 con 0x7fdddc0af4d0 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fddd8ff9700 1 -- 172.21.15.89:0/2925332748 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4013b20 con 0x7fdddc0af4d0 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.088+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc1f7e10 con 0x7fdddc0af4d0 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.089+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc0753f0 con 0x7fdddc0af4d0 2023-11-24T06:19:17.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.090+0000 7fddd8ff9700 1 -- 172.21.15.89:0/2925332748 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fddc4013c80 con 0x7fdddc0af4d0 2023-11-24T06:19:17.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.090+0000 7fddd8ff9700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fddbc06aed0 0x7fddbc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.090+0000 7fddd8ff9700 1 -- 172.21.15.89:0/2925332748 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddc40890e0 con 0x7fdddc0af4d0 2023-11-24T06:19:17.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.090+0000 7fdddb7fe700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fddbc06aed0 0x7fddbc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.091+0000 7fdddb7fe700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fddbc06aed0 0x7fddbc06d390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fddcc007fd0 tx=0x7fddcc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.095+0000 7fddd8ff9700 1 -- 172.21.15.89:0/2925332748 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddc40541a0 con 0x7fdddc0af4d0 2023-11-24T06:19:17.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.287+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdddc067f60 con 0x7fdddc0af4d0 2023-11-24T06:19:17.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.288+0000 7fddd8ff9700 1 -- 172.21.15.89:0/2925332748 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7fddc40577c0 con 0x7fdddc0af4d0 2023-11-24T06:19:17.292 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:17.292 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fddbc06aed0 msgr2=0x7fddbc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fddbc06aed0 0x7fddbc06d390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fddcc007fd0 tx=0x7fddcc009040 comp rx=0 tx=0).stop 2023-11-24T06:19:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 msgr2=0x7fdddc1f7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc1f7880 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fddc400e840 tx=0x7fddc400eb50 comp rx=0 tx=0).stop 2023-11-24T06:19:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 shutdown_connections 2023-11-24T06:19:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fddbc06aed0 0x7fddbc06d390 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdddc15e080 0x7fdddc0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdddc06c710 0x7fdddc14da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 --2- 172.21.15.89:0/2925332748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdddc0af4d0 0x7fdddc1f7880 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 >> 172.21.15.89:0/2925332748 conn(0x7fdddc071b90 msgr2=0x7fdddc06e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 shutdown_connections 2023-11-24T06:19:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.290+0000 7fdde1d13700 1 -- 172.21.15.89:0/2925332748 wait complete. 2023-11-24T06:19:17.306 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:17.307 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:17.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.426+0000 7fa8550ee700 1 Processor -- start 2023-11-24T06:19:17.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.426+0000 7fa8550ee700 1 -- start start 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa8550ee700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa85015dec0 0x7fa85006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa8550ee700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8500ce4c0 0x7fa85006f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa8550ee700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa85015e2a0 0x7fa85006f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa8550ee700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8500cfc60 con 0x7fa85015e2a0 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa8550ee700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8500cf4d0 con 0x7fa85015dec0 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa8550ee700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8500ce950 con 0x7fa8500ce4c0 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa84f59e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa85015e2a0 0x7fa85006f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa84f59e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa85015e2a0 0x7fa85006f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35990/0 (socket says 172.21.15.89:35990) 2023-11-24T06:19:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa84f59e700 1 -- 172.21.15.89:0/3646012068 learned_addr learned my addr 172.21.15.89:0/3646012068 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa84e59c700 1 --2- 172.21.15.89:0/3646012068 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8500ce4c0 0x7fa85006f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.427+0000 7fa84ed9d700 1 --2- 172.21.15.89:0/3646012068 >> v2:172.21.15.123:3300/0 conn(0x7fa85015dec0 0x7fa85006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84f59e700 1 -- 172.21.15.89:0/3646012068 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8500ce4c0 msgr2=0x7fa85006f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84f59e700 1 --2- 172.21.15.89:0/3646012068 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8500ce4c0 0x7fa85006f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84f59e700 1 -- 172.21.15.89:0/3646012068 >> v2:172.21.15.123:3300/0 conn(0x7fa85015dec0 msgr2=0x7fa85006ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84f59e700 1 --2- 172.21.15.89:0/3646012068 >> v2:172.21.15.123:3300/0 conn(0x7fa85015dec0 0x7fa85006ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84f59e700 1 -- 172.21.15.89:0/3646012068 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8500cec30 con 0x7fa85015e2a0 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84ed9d700 1 --2- 172.21.15.89:0/3646012068 >> v2:172.21.15.123:3300/0 conn(0x7fa85015dec0 0x7fa85006ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84f59e700 1 --2- 172.21.15.89:0/3646012068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015e2a0 0x7fa85006f770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa8500cf760 tx=0x7fa84800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804fe1167747a184 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84dd9b700 1 -- 172.21.15.89:0/3646012068 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84800e070 con 0x7fa85015e2a0 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84dd9b700 1 -- 172.21.15.89:0/3646012068 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84800b9c0 con 0x7fa85015e2a0 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa84dd9b700 1 -- 172.21.15.89:0/3646012068 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84800bd70 con 0x7fa85015e2a0 2023-11-24T06:19:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa8550ee700 1 -- 172.21.15.89:0/3646012068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015e2a0 msgr2=0x7fa85006f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa8550ee700 1 --2- 172.21.15.89:0/3646012068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015e2a0 0x7fa85006f770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa8500cf760 tx=0x7fa84800b330 comp rx=0 tx=0).stop 2023-11-24T06:19:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa8550ee700 1 -- 172.21.15.89:0/3646012068 shutdown_connections 2023-11-24T06:19:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa8550ee700 1 --2- 172.21.15.89:0/3646012068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015e2a0 0x7fa85006f770 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa8550ee700 1 --2- 172.21.15.89:0/3646012068 >> v2:172.21.15.123:3300/0 conn(0x7fa85015dec0 0x7fa85006ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa8550ee700 1 --2- 172.21.15.89:0/3646012068 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8500ce4c0 0x7fa85006f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.428+0000 7fa8550ee700 1 -- 172.21.15.89:0/3646012068 >> 172.21.15.89:0/3646012068 conn(0x7fa8500730a0 msgr2=0x7fa85009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.429+0000 7fa8550ee700 1 -- 172.21.15.89:0/3646012068 shutdown_connections 2023-11-24T06:19:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.429+0000 7fa8550ee700 1 -- 172.21.15.89:0/3646012068 wait complete. 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.430+0000 7fa8550ee700 1 Processor -- start 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.430+0000 7fa8550ee700 1 -- start start 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa8550ee700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015dec0 0x7fa8501512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa8550ee700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa85015e2a0 0x7fa85015e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa8550ee700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8501f36d0 0x7fa8501f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa8550ee700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8501600e0 con 0x7fa85015dec0 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa8550ee700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85015ebf0 con 0x7fa8501f36d0 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa8550ee700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8501615a0 con 0x7fa85015e2a0 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015dec0 0x7fa8501512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015dec0 0x7fa8501512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35992/0 (socket says 172.21.15.89:35992) 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 -- 172.21.15.89:0/121248761 learned_addr learned my addr 172.21.15.89:0/121248761 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84e59c700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa85015e2a0 0x7fa85015e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84f59e700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8501f36d0 0x7fa8501f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 -- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa85015e2a0 msgr2=0x7fa85015e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa85015e2a0 0x7fa85015e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 -- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8501f36d0 msgr2=0x7fa8501f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8501f36d0 0x7fa8501f3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 -- 172.21.15.89:0/121248761 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa848009b30 con 0x7fa85015dec0 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84e59c700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa85015e2a0 0x7fa85015e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa84ed9d700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015dec0 0x7fa8501512f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa83800e870 tx=0x7fa83800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa8477fe700 1 -- 172.21.15.89:0/121248761 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838013610 con 0x7fa85015dec0 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.431+0000 7fa8477fe700 1 -- 172.21.15.89:0/121248761 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838013770 con 0x7fa85015dec0 2023-11-24T06:19:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.432+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8501612c0 con 0x7fa85015dec0 2023-11-24T06:19:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.432+0000 7fa8477fe700 1 -- 172.21.15.89:0/121248761 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838013b20 con 0x7fa85015dec0 2023-11-24T06:19:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.432+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85015fd00 con 0x7fa85015dec0 2023-11-24T06:19:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.432+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa850068fb0 con 0x7fa85015dec0 2023-11-24T06:19:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.433+0000 7fa8477fe700 1 -- 172.21.15.89:0/121248761 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa838013c80 con 0x7fa85015dec0 2023-11-24T06:19:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.433+0000 7fa8477fe700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa83006aed0 0x7fa83006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.433+0000 7fa8477fe700 1 -- 172.21.15.89:0/121248761 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa838089120 con 0x7fa85015dec0 2023-11-24T06:19:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.433+0000 7fa84e59c700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa83006aed0 0x7fa83006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.434+0000 7fa84e59c700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa83006aed0 0x7fa83006d390 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa8500b10b0 tx=0x7fa840008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.438+0000 7fa8477fe700 1 -- 172.21.15.89:0/121248761 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa838054260 con 0x7fa85015dec0 2023-11-24T06:19:17.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.630+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8500b2d90 con 0x7fa85015dec0 2023-11-24T06:19:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.631+0000 7fa8477fe700 1 -- 172.21.15.89:0/121248761 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa838057880 con 0x7fa85015dec0 2023-11-24T06:19:17.634 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa83006aed0 msgr2=0x7fa83006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa83006aed0 0x7fa83006d390 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa8500b10b0 tx=0x7fa840008040 comp rx=0 tx=0).stop 2023-11-24T06:19:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015dec0 msgr2=0x7fa8501512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015dec0 0x7fa8501512f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa83800e870 tx=0x7fa83800eb80 comp rx=0 tx=0).stop 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 shutdown_connections 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa83006aed0 0x7fa83006d390 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8501f36d0 0x7fa8501f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85015dec0 0x7fa8501512f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 --2- 172.21.15.89:0/121248761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa85015e2a0 0x7fa85015e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 >> 172.21.15.89:0/121248761 conn(0x7fa8500730a0 msgr2=0x7fa85015c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 shutdown_connections 2023-11-24T06:19:17.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.633+0000 7fa8550ee700 1 -- 172.21.15.89:0/121248761 wait complete. 2023-11-24T06:19:17.648 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:17.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.768+0000 7f2fd43e3700 1 Processor -- start 2023-11-24T06:19:17.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.768+0000 7f2fd43e3700 1 -- start start 2023-11-24T06:19:17.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd43e3700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2fcc0cc540 0x7f2fcc06e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd43e3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2fcc06eb20 0x7f2fcc0b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd43e3700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc06ef50 con 0x7f2fcc15a620 2023-11-24T06:19:17.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd43e3700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc0b2f20 con 0x7f2fcc06eb20 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd43e3700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc15aa00 con 0x7f2fcc0cc540 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd2980700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2fcc06eb20 0x7f2fcc0b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd2980700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2fcc06eb20 0x7f2fcc0b3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36004/0 (socket says 172.21.15.89:36004) 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd2980700 1 -- 172.21.15.89:0/246850309 learned_addr learned my addr 172.21.15.89:0/246850309 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd197e700 1 --2- 172.21.15.89:0/246850309 >> v2:172.21.15.123:3300/0 conn(0x7f2fcc0cc540 0x7f2fcc06e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd2980700 1 -- 172.21.15.89:0/246850309 >> v2:172.21.15.123:3300/0 conn(0x7f2fcc0cc540 msgr2=0x7f2fcc06e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd2980700 1 --2- 172.21.15.89:0/246850309 >> v2:172.21.15.123:3300/0 conn(0x7f2fcc0cc540 0x7f2fcc06e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd2980700 1 -- 172.21.15.89:0/246850309 >> v1:172.21.15.123:6789/0 conn(0x7f2fcc15a620 legacy=0x7f2fcc0ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd2980700 1 -- 172.21.15.89:0/246850309 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0ce960 con 0x7f2fcc06eb20 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd2980700 1 --2- 172.21.15.89:0/246850309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc0b3350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc075600 tx=0x7f2fc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87974a481cc0a770 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.769+0000 7f2fd117d700 1 -- 172.21.15.89:0/246850309 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400e070 con 0x7f2fcc06eb20 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd117d700 1 -- 172.21.15.89:0/246850309 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc400b910 con 0x7f2fcc06eb20 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd117d700 1 -- 172.21.15.89:0/246850309 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400bcc0 con 0x7f2fcc06eb20 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd43e3700 1 -- 172.21.15.89:0/246850309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 msgr2=0x7f2fcc0b3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/246850309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc0b3350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc075600 tx=0x7f2fc400b330 comp rx=0 tx=0).stop 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd43e3700 1 -- 172.21.15.89:0/246850309 shutdown_connections 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/246850309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc0b3350 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/246850309 >> v2:172.21.15.123:3300/0 conn(0x7f2fcc0cc540 0x7f2fcc06e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd43e3700 1 -- 172.21.15.89:0/246850309 >> 172.21.15.89:0/246850309 conn(0x7f2fcc0b2050 msgr2=0x7f2fcc0b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:17.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd43e3700 1 -- 172.21.15.89:0/246850309 shutdown_connections 2023-11-24T06:19:17.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.770+0000 7f2fd43e3700 1 -- 172.21.15.89:0/246850309 wait complete. 2023-11-24T06:19:17.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.771+0000 7f2fd43e3700 1 Processor -- start 2023-11-24T06:19:17.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.771+0000 7f2fd43e3700 1 -- start start 2023-11-24T06:19:17.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd43e3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd43e3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fcc0cc540 0x7f2fcc149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd43e3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fcc15a620 0x7f2fcc150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd43e3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15c650 con 0x7f2fcc06eb20 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd43e3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15c8f0 con 0x7f2fcc0cc540 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd43e3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15cbd0 con 0x7f2fcc15a620 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd217f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd217f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc148e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36006/0 (socket says 172.21.15.89:36006) 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd217f700 1 -- 172.21.15.89:0/3098860411 learned_addr learned my addr 172.21.15.89:0/3098860411 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd197e700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fcc0cc540 0x7f2fcc149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd2980700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fcc15a620 0x7f2fcc150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd217f700 1 -- 172.21.15.89:0/3098860411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fcc15a620 msgr2=0x7f2fcc150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd217f700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fcc15a620 0x7f2fcc150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd217f700 1 -- 172.21.15.89:0/3098860411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fcc0cc540 msgr2=0x7f2fcc149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd217f700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fcc0cc540 0x7f2fcc149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.772+0000 7f2fd217f700 1 -- 172.21.15.89:0/3098860411 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4009b30 con 0x7f2fcc06eb20 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.773+0000 7f2fd217f700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc148e10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800c450 tx=0x7f2fc800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.773+0000 7f2fc2ffd700 1 -- 172.21.15.89:0/3098860411 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80052d0 con 0x7f2fcc06eb20 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.773+0000 7f2fc2ffd700 1 -- 172.21.15.89:0/3098860411 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc8005430 con 0x7f2fcc06eb20 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.773+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc150530 con 0x7f2fcc06eb20 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.773+0000 7f2fc2ffd700 1 -- 172.21.15.89:0/3098860411 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80038c0 con 0x7f2fcc06eb20 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.773+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc14a2e0 con 0x7f2fcc06eb20 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.774+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc068d90 con 0x7f2fcc06eb20 2023-11-24T06:19:17.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.774+0000 7f2fc2ffd700 1 -- 172.21.15.89:0/3098860411 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2fc8003a20 con 0x7f2fcc06eb20 2023-11-24T06:19:17.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.774+0000 7f2fc2ffd700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2fb006aed0 0x7f2fb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:17.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.775+0000 7f2fc2ffd700 1 -- 172.21.15.89:0/3098860411 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fc8088980 con 0x7f2fcc06eb20 2023-11-24T06:19:17.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.775+0000 7f2fd197e700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2fb006aed0 0x7f2fb006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:17.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.775+0000 7f2fd197e700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2fb006aed0 0x7f2fb006d390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0af850 tx=0x7f2fbc009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:17.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.779+0000 7f2fc2ffd700 1 -- 172.21.15.89:0/3098860411 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fc8053a30 con 0x7f2fcc06eb20 2023-11-24T06:19:17.858 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:19:17.861 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:19:17.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.979+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2fcc068fb0 con 0x7f2fcc06eb20 2023-11-24T06:19:17.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.980+0000 7f2fc2ffd700 1 -- 172.21.15.89:0/3098860411 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2fc8017020 con 0x7f2fcc06eb20 2023-11-24T06:19:17.983 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:17.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2fb006aed0 msgr2=0x7f2fb006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2fb006aed0 0x7f2fb006d390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0af850 tx=0x7f2fbc009210 comp rx=0 tx=0).stop 2023-11-24T06:19:17.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 msgr2=0x7f2fcc148e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:17.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc148e10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800c450 tx=0x7f2fc800c760 comp rx=0 tx=0).stop 2023-11-24T06:19:17.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 shutdown_connections 2023-11-24T06:19:17.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fcc15a620 0x7f2fcc150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fcc06eb20 0x7f2fcc148e10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2fb006aed0 0x7f2fb006d390 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 --2- 172.21.15.89:0/3098860411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fcc0cc540 0x7f2fcc149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:17.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 >> 172.21.15.89:0/3098860411 conn(0x7f2fcc0b2050 msgr2=0x7f2fcc158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:17.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 shutdown_connections 2023-11-24T06:19:17.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:17.982+0000 7f2fd43e3700 1 -- 172.21.15.89:0/3098860411 wait complete. 2023-11-24T06:19:17.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:18.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.117+0000 7f8793545700 1 Processor -- start 2023-11-24T06:19:18.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.117+0000 7f8793545700 1 -- start start 2023-11-24T06:19:18.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f8793545700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c15e160 0x7f878c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f8793545700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f878c0746a0 0x7f878c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f8793545700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c0b3c60 con 0x7f878c0afc90 2023-11-24T06:19:18.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f8793545700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c0b4500 con 0x7f878c0746a0 2023-11-24T06:19:18.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f8793545700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0b58c0 con 0x7f878c15e160 2023-11-24T06:19:18.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f8791ae2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f878c0746a0 0x7f878c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f87912e1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c15e160 0x7f878c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f8790ae0700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f878c0afc90 0x7f878c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:43340/0 (socket says 172.21.15.89:43340) 2023-11-24T06:19:18.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.118+0000 7f8790ae0700 1 -- 172.21.15.89:0/1454229045 learned_addr learned my addr 172.21.15.89:0/1454229045 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:18.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.119+0000 7f8791ae2700 1 -- 172.21.15.89:0/1454229045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c15e160 msgr2=0x7f878c0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:18.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.119+0000 7f8791ae2700 1 --2- 172.21.15.89:0/1454229045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c15e160 0x7f878c0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.119+0000 7f8791ae2700 1 -- 172.21.15.89:0/1454229045 >> v1:172.21.15.123:6789/0 conn(0x7f878c0afc90 legacy=0x7f878c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.119+0000 7f8791ae2700 1 -- 172.21.15.89:0/1454229045 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c06bdc0 con 0x7f878c0746a0 2023-11-24T06:19:18.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.119+0000 7f8791ae2700 1 --2- 172.21.15.89:0/1454229045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0746a0 0x7f878c06d930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8784009d90 tx=0x7f878400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c87a5d0190fa0510 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.119+0000 7f8783fff700 1 -- 172.21.15.89:0/1454229045 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878400e070 con 0x7f878c0746a0 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.119+0000 7f8783fff700 1 -- 172.21.15.89:0/1454229045 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878400bb20 con 0x7f878c0746a0 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.119+0000 7f8783fff700 1 -- 172.21.15.89:0/1454229045 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784012430 con 0x7f878c0746a0 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.120+0000 7f8793545700 1 -- 172.21.15.89:0/1454229045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0746a0 msgr2=0x7f878c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.120+0000 7f8793545700 1 --2- 172.21.15.89:0/1454229045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0746a0 0x7f878c06d930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8784009d90 tx=0x7f878400b330 comp rx=0 tx=0).stop 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.120+0000 7f8793545700 1 -- 172.21.15.89:0/1454229045 shutdown_connections 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.120+0000 7f8793545700 1 --2- 172.21.15.89:0/1454229045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0746a0 0x7f878c06d930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.120+0000 7f8793545700 1 --2- 172.21.15.89:0/1454229045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c15e160 0x7f878c0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.120+0000 7f8793545700 1 -- 172.21.15.89:0/1454229045 >> 172.21.15.89:0/1454229045 conn(0x7f878c0d0bf0 msgr2=0x7f878c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.120+0000 7f8793545700 1 -- 172.21.15.89:0/1454229045 shutdown_connections 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.120+0000 7f8793545700 1 -- 172.21.15.89:0/1454229045 wait complete. 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.121+0000 7f8793545700 1 Processor -- start 2023-11-24T06:19:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.121+0000 7f8793545700 1 -- start start 2023-11-24T06:19:18.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8793545700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c0746a0 0x7f878c158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8793545700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f878c0afc90 0x7f878c160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8793545700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c15e160 0x7f878c160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8793545700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c160200 con 0x7f878c0afc90 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8793545700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c15f5f0 con 0x7f878c15e160 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8793545700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c15f8d0 con 0x7f878c0746a0 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f87912e1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c0746a0 0x7f878c158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f87912e1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c0746a0 0x7f878c158890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:44896/0 (socket says 172.21.15.89:44896) 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f87912e1700 1 -- 172.21.15.89:0/1152226455 learned_addr learned my addr 172.21.15.89:0/1152226455 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8790ae0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f878c0afc90 0x7f878c160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8791ae2700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c15e160 0x7f878c160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8790ae0700 1 -- 172.21.15.89:0/1152226455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c0746a0 msgr2=0x7f878c158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8790ae0700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c0746a0 0x7f878c158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8790ae0700 1 -- 172.21.15.89:0/1152226455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c15e160 msgr2=0x7f878c160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8790ae0700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c15e160 0x7f878c160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.122+0000 7f8790ae0700 1 -- 172.21.15.89:0/1152226455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784009a70 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.123+0000 7f8790ae0700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f878c0afc90 0x7f878c160a60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f878800c500 tx=0x7f878800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.123+0000 7f8781ffb700 1 -- 172.21.15.89:0/1152226455 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788005d70 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.123+0000 7f8781ffb700 1 -- 172.21.15.89:0/1152226455 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87880070d0 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.123+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c15fbb0 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.123+0000 7f8781ffb700 1 -- 172.21.15.89:0/1152226455 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788007440 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.123+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c158dd0 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.124+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c074e60 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.124+0000 7f8781ffb700 1 -- 172.21.15.89:0/1152226455 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f87880075a0 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.125+0000 7f8781ffb700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.125+0000 7f8781ffb700 1 -- 172.21.15.89:0/1152226455 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8788088730 con 0x7f878c0afc90 2023-11-24T06:19:18.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.125+0000 7f87912e1700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.125+0000 7f87912e1700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f877c009c00 tx=0x7f877c009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.129+0000 7f8781ffb700 1 -- 172.21.15.89:0/1152226455 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8788053920 con 0x7f878c0afc90 2023-11-24T06:19:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.323+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f878c068fb0 con 0x7f878c0afc90 2023-11-24T06:19:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.324+0000 7f8781ffb700 1 -- 172.21.15.89:0/1152226455 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8788056f40 con 0x7f878c0afc90 2023-11-24T06:19:18.327 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 msgr2=0x7f877006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f877c009c00 tx=0x7f877c009210 comp rx=0 tx=0).stop 2023-11-24T06:19:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f878c0afc90 msgr2=0x7f878c160a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f878c0afc90 0x7f878c160a60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f878800c500 tx=0x7f878800c810 comp rx=0 tx=0).stop 2023-11-24T06:19:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 shutdown_connections 2023-11-24T06:19:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f877006aed0 0x7f877006d390 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c15e160 0x7f878c160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878c0746a0 0x7f878c158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 --2- 172.21.15.89:0/1152226455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f878c0afc90 0x7f878c160a60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 >> 172.21.15.89:0/1152226455 conn(0x7f878c0d0bf0 msgr2=0x7f878c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 shutdown_connections 2023-11-24T06:19:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.326+0000 7f8793545700 1 -- 172.21.15.89:0/1152226455 wait complete. 2023-11-24T06:19:18.415 INFO:tasks.cephfs_test_runner:Starting test: test_volume_create (tasks.cephfs.test_volumes.TestVolumes) That the volume can be created and then cleans up 2023-11-24T06:19:18.415 INFO:tasks.cephfs_test_runner:test_volume_create (tasks.cephfs.test_volumes.TestVolumes) 2023-11-24T06:19:18.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:18.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.532+0000 7f9776deb700 1 Processor -- start 2023-11-24T06:19:18.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f9776deb700 1 -- start start 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f9776deb700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97700b0f80 0x7f977015a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f9776deb700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f97700cd770 0x7f977015a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f9776deb700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97700b2bd0 con 0x7f97700b0ba0 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f9776deb700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97700b1fc0 con 0x7f97700b0f80 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f9776deb700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97700b22a0 con 0x7f97700cd770 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f976ffff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97700b0f80 0x7f977015a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f9774b87700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f97700b0ba0 0x7f97700af5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39532/0 (socket says 172.21.15.89:39532) 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.533+0000 7f9774b87700 1 -- 172.21.15.89:0/4042796892 learned_addr learned my addr 172.21.15.89:0/4042796892 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f9775388700 1 --2- 172.21.15.89:0/4042796892 >> v2:172.21.15.123:3300/0 conn(0x7f97700cd770 0x7f977015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976ffff700 1 -- 172.21.15.89:0/4042796892 >> v2:172.21.15.123:3300/0 conn(0x7f97700cd770 msgr2=0x7f977015a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976ffff700 1 --2- 172.21.15.89:0/4042796892 >> v2:172.21.15.123:3300/0 conn(0x7f97700cd770 0x7f977015a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976ffff700 1 -- 172.21.15.89:0/4042796892 >> v1:172.21.15.89:6789/0 conn(0x7f97700b0ba0 legacy=0x7f97700af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976ffff700 1 -- 172.21.15.89:0/4042796892 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97700b2580 con 0x7f97700b0f80 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976f7fe700 1 -- 172.21.15.89:0/4042796892 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205932703 0 0) 0x7f97700b2bd0 con 0x7f97700b0ba0 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976ffff700 1 --2- 172.21.15.89:0/4042796892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f977015a440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f97700b2b40 tx=0x7f975c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad31ab1cfaee0c3a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976f7fe700 1 -- 172.21.15.89:0/4042796892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c011830 con 0x7f97700b0f80 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976f7fe700 1 -- 172.21.15.89:0/4042796892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c011990 con 0x7f97700b0f80 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.534+0000 7f976f7fe700 1 -- 172.21.15.89:0/4042796892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c011d20 con 0x7f97700b0f80 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.535+0000 7f9776deb700 1 -- 172.21.15.89:0/4042796892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 msgr2=0x7f977015a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.535+0000 7f9776deb700 1 --2- 172.21.15.89:0/4042796892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f977015a440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f97700b2b40 tx=0x7f975c009b20 comp rx=0 tx=0).stop 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.535+0000 7f9776deb700 1 -- 172.21.15.89:0/4042796892 shutdown_connections 2023-11-24T06:19:18.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.535+0000 7f9776deb700 1 --2- 172.21.15.89:0/4042796892 >> v2:172.21.15.123:3300/0 conn(0x7f97700cd770 0x7f977015a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.535+0000 7f9776deb700 1 --2- 172.21.15.89:0/4042796892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f977015a440 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.535+0000 7f9776deb700 1 -- 172.21.15.89:0/4042796892 >> 172.21.15.89:0/4042796892 conn(0x7f97700b57b0 msgr2=0x7f97700b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:18.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.535+0000 7f9776deb700 1 -- 172.21.15.89:0/4042796892 shutdown_connections 2023-11-24T06:19:18.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.535+0000 7f9776deb700 1 -- 172.21.15.89:0/4042796892 wait complete. 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.536+0000 7f9776deb700 1 Processor -- start 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.536+0000 7f9776deb700 1 -- start start 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f9776deb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97700b0ba0 0x7f9770152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f9776deb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f9770154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f9776deb700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97700cd770 0x7f977015a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f9776deb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9770152d90 con 0x7f97700b0f80 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f9776deb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97701547b0 con 0x7f97700b0ba0 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f9776deb700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97701549e0 con 0x7f97700cd770 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f976ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f9770154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f976ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f9770154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36036/0 (socket says 172.21.15.89:36036) 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.537+0000 7f976ffff700 1 -- 172.21.15.89:0/1893915185 learned_addr learned my addr 172.21.15.89:0/1893915185 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f9774b87700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97700b0ba0 0x7f9770152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f9775388700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97700cd770 0x7f977015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976ffff700 1 -- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97700cd770 msgr2=0x7f977015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976ffff700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97700cd770 0x7f977015a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976ffff700 1 -- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97700b0ba0 msgr2=0x7f9770152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976ffff700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97700b0ba0 0x7f9770152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976ffff700 1 -- 172.21.15.89:0/1893915185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c0094a0 con 0x7f97700b0f80 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f9775388700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97700cd770 0x7f977015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976ffff700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f9770154270 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f975c0090b0 tx=0x7f975c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976d7fa700 1 -- 172.21.15.89:0/1893915185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c0038a0 con 0x7f97700b0f80 2023-11-24T06:19:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976d7fa700 1 -- 172.21.15.89:0/1893915185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c01f070 con 0x7f97700b0f80 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9770154cc0 con 0x7f97700b0f80 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f976d7fa700 1 -- 172.21.15.89:0/1893915185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c003c20 con 0x7f97700b0f80 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.538+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9770155280 con 0x7f97700b0f80 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.539+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9770067f60 con 0x7f97700b0f80 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.539+0000 7f976d7fa700 1 -- 172.21.15.89:0/1893915185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f975c01a430 con 0x7f97700b0f80 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.540+0000 7f976d7fa700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f975406aed0 0x7f975406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.540+0000 7f976d7fa700 1 -- 172.21.15.89:0/1893915185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f975c021070 con 0x7f97700b0f80 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.540+0000 7f9774b87700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f975406aed0 0x7f975406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.540+0000 7f9774b87700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f975406aed0 0x7f975406d390 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9764000f60 tx=0x7f9764007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.544+0000 7f976d7fa700 1 -- 172.21.15.89:0/1893915185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f975c056110 con 0x7f97700b0f80 2023-11-24T06:19:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.735+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9770071f00 con 0x7f97700b0f80 2023-11-24T06:19:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.736+0000 7f976d7fa700 1 -- 172.21.15.89:0/1893915185 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f975c059730 con 0x7f97700b0f80 2023-11-24T06:19:18.739 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:18.740 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f975406aed0 msgr2=0x7f975406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f975406aed0 0x7f975406d390 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9764000f60 tx=0x7f9764007be0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 msgr2=0x7f9770154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f9770154270 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f975c0090b0 tx=0x7f975c004750 comp rx=0 tx=0).stop 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 shutdown_connections 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97700cd770 0x7f977015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f975406aed0 0x7f975406d390 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97700b0ba0 0x7f9770152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 --2- 172.21.15.89:0/1893915185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97700b0f80 0x7f9770154270 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 >> 172.21.15.89:0/1893915185 conn(0x7f97700b57b0 msgr2=0x7f977006df50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 shutdown_connections 2023-11-24T06:19:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.739+0000 7f9776deb700 1 -- 172.21.15.89:0/1893915185 wait complete. 2023-11-24T06:19:18.755 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:18.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:18.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.875+0000 7f226784a700 1 Processor -- start 2023-11-24T06:19:18.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.875+0000 7f226784a700 1 -- start start 2023-11-24T06:19:18.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f226784a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f226015be20 0x7f22600af790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f226784a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f22600b4e00 0x7f22600b5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f226784a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22600b4970 con 0x7f22600cd840 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f226784a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22600cfed0 con 0x7f226015be20 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f226784a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22600b4360 con 0x7f22600b4e00 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f22655e6700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f226015be20 0x7f22600af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f22655e6700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f226015be20 0x7f22600af790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36042/0 (socket says 172.21.15.89:36042) 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f22655e6700 1 -- 172.21.15.89:0/3575150137 learned_addr learned my addr 172.21.15.89:0/3575150137 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f2264de5700 1 --2- 172.21.15.89:0/3575150137 >> v2:172.21.15.123:3300/0 conn(0x7f22600b4e00 0x7f22600b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f22655e6700 1 -- 172.21.15.89:0/3575150137 >> v2:172.21.15.123:3300/0 conn(0x7f22600b4e00 msgr2=0x7f22600b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f22655e6700 1 --2- 172.21.15.89:0/3575150137 >> v2:172.21.15.123:3300/0 conn(0x7f22600b4e00 0x7f22600b5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f22655e6700 1 -- 172.21.15.89:0/3575150137 >> v1:172.21.15.123:6789/0 conn(0x7f22600cd840 legacy=0x7f226015c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f22655e6700 1 -- 172.21.15.89:0/3575150137 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226006f940 con 0x7f226015be20 2023-11-24T06:19:18.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.876+0000 7f22655e6700 1 --2- 172.21.15.89:0/3575150137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f226015be20 0x7f22600af790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f22600af550 tx=0x7f225c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900d22d7dca2f9be server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f2257fff700 1 -- 172.21.15.89:0/3575150137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c011830 con 0x7f226015be20 2023-11-24T06:19:18.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f2257fff700 1 -- 172.21.15.89:0/3575150137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c011990 con 0x7f226015be20 2023-11-24T06:19:18.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f2257fff700 1 -- 172.21.15.89:0/3575150137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c011d20 con 0x7f226015be20 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f226784a700 1 -- 172.21.15.89:0/3575150137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f226015be20 msgr2=0x7f22600af790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f226784a700 1 --2- 172.21.15.89:0/3575150137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f226015be20 0x7f22600af790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f22600af550 tx=0x7f225c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f226784a700 1 -- 172.21.15.89:0/3575150137 shutdown_connections 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f226784a700 1 --2- 172.21.15.89:0/3575150137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f226015be20 0x7f22600af790 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f226784a700 1 --2- 172.21.15.89:0/3575150137 >> v2:172.21.15.123:3300/0 conn(0x7f22600b4e00 0x7f22600b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f226784a700 1 -- 172.21.15.89:0/3575150137 >> 172.21.15.89:0/3575150137 conn(0x7f22600b2050 msgr2=0x7f22600b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f226784a700 1 -- 172.21.15.89:0/3575150137 shutdown_connections 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.877+0000 7f226784a700 1 -- 172.21.15.89:0/3575150137 wait complete. 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.879+0000 7f226784a700 1 Processor -- start 2023-11-24T06:19:18.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.879+0000 7f226784a700 1 -- start start 2023-11-24T06:19:18.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f226784a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22600b4e00 0x7f226015c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f226784a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22600cd840 0x7f2260154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f226784a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226015be20 0x7f2260148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f226784a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226006d650 con 0x7f22600cd840 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f226784a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260155700 con 0x7f226015be20 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f226784a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226006e2c0 con 0x7f22600b4e00 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22600cd840 0x7f2260154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22600cd840 0x7f2260154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36050/0 (socket says 172.21.15.89:36050) 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 -- 172.21.15.89:0/1400785414 learned_addr learned my addr 172.21.15.89:0/1400785414 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f22655e6700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22600b4e00 0x7f226015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2265de7700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226015be20 0x7f2260148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 -- 172.21.15.89:0/1400785414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22600b4e00 msgr2=0x7f226015c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22600b4e00 0x7f226015c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 -- 172.21.15.89:0/1400785414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226015be20 msgr2=0x7f2260148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:18.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226015be20 0x7f2260148d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:18.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 -- 172.21.15.89:0/1400785414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c009550 con 0x7f22600cd840 2023-11-24T06:19:18.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2264de5700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22600cd840 0x7f2260154f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f225000e9e0 tx=0x7f225000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2255ffb700 1 -- 172.21.15.89:0/1400785414 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250009710 con 0x7f22600cd840 2023-11-24T06:19:18.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2255ffb700 1 -- 172.21.15.89:0/1400785414 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225000ee80 con 0x7f22600cd840 2023-11-24T06:19:18.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f2255ffb700 1 -- 172.21.15.89:0/1400785414 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225000ca50 con 0x7f22600cd840 2023-11-24T06:19:18.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.880+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226015c740 con 0x7f22600cd840 2023-11-24T06:19:18.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.881+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2260155490 con 0x7f22600cd840 2023-11-24T06:19:18.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.882+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2260068d90 con 0x7f22600cd840 2023-11-24T06:19:18.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.882+0000 7f2255ffb700 1 -- 172.21.15.89:0/1400785414 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f225000cbb0 con 0x7f22600cd840 2023-11-24T06:19:18.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.882+0000 7f2255ffb700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f224406aed0 0x7f224406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:18.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.882+0000 7f2255ffb700 1 -- 172.21.15.89:0/1400785414 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2250089520 con 0x7f22600cd840 2023-11-24T06:19:18.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.882+0000 7f22655e6700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f224406aed0 0x7f224406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:18.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.883+0000 7f22655e6700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f224406aed0 0x7f224406d390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f225c000c00 tx=0x7f225c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:18.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:18.887+0000 7f2255ffb700 1 -- 172.21.15.89:0/1400785414 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2250054660 con 0x7f22600cd840 2023-11-24T06:19:19.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.076+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2260068fb0 con 0x7f22600cd840 2023-11-24T06:19:19.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.076+0000 7f2255ffb700 1 -- 172.21.15.89:0/1400785414 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2250057c80 con 0x7f22600cd840 2023-11-24T06:19:19.079 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:19.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f224406aed0 msgr2=0x7f224406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f224406aed0 0x7f224406d390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f225c000c00 tx=0x7f225c008030 comp rx=0 tx=0).stop 2023-11-24T06:19:19.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22600cd840 msgr2=0x7f2260154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22600cd840 0x7f2260154f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f225000e9e0 tx=0x7f225000ecf0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 shutdown_connections 2023-11-24T06:19:19.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f224406aed0 0x7f224406d390 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226015be20 0x7f2260148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22600b4e00 0x7f226015c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 --2- 172.21.15.89:0/1400785414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22600cd840 0x7f2260154f00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 >> 172.21.15.89:0/1400785414 conn(0x7f22600b2050 msgr2=0x7f22600cc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:19.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 shutdown_connections 2023-11-24T06:19:19.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.078+0000 7f226784a700 1 -- 172.21.15.89:0/1400785414 wait complete. 2023-11-24T06:19:19.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:19.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.213+0000 7f88aca80700 1 Processor -- start 2023-11-24T06:19:19.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.213+0000 7f88aca80700 1 -- start start 2023-11-24T06:19:19.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.214+0000 7f88aca80700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88a815e160 0x7f88a80b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.214+0000 7f88aca80700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a80b3c60 con 0x7f88a80746a0 2023-11-24T06:19:19.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.214+0000 7f88aca80700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a80b58c0 con 0x7f88a80afc90 2023-11-24T06:19:19.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.214+0000 7f88aca80700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a80b4500 con 0x7f88a815e160 2023-11-24T06:19:19.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.214+0000 7f88a659c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88a815e160 0x7f88a80b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.214+0000 7f88a6d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f88a80746a0 0x7f88a806d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39562/0 (socket says 172.21.15.89:39562) 2023-11-24T06:19:19.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.214+0000 7f88a6d9d700 1 -- 172.21.15.89:0/3611168328 learned_addr learned my addr 172.21.15.89:0/3611168328 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:19.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.215+0000 7f88a659c700 1 -- 172.21.15.89:0/3611168328 >> v1:172.21.15.123:6789/0 conn(0x7f88a80afc90 legacy=0x7f88a8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:19.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.215+0000 7f88a659c700 1 -- 172.21.15.89:0/3611168328 >> v1:172.21.15.89:6789/0 conn(0x7f88a80746a0 legacy=0x7f88a806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.215+0000 7f88a659c700 1 -- 172.21.15.89:0/3611168328 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80b0140 con 0x7f88a815e160 2023-11-24T06:19:19.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.215+0000 7f88a559a700 1 -- 172.21.15.89:0/3611168328 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703863814 0 0) 0x7f88a80b3c60 con 0x7f88a80746a0 2023-11-24T06:19:19.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.215+0000 7f88a659c700 1 --2- 172.21.15.89:0/3611168328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a80b5da0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f889c009840 tx=0x7f889c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecaa8d4fafa1d65 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.215+0000 7f88a559a700 1 -- 172.21.15.89:0/3611168328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c004230 con 0x7f88a815e160 2023-11-24T06:19:19.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.215+0000 7f88a559a700 1 -- 172.21.15.89:0/3611168328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c004390 con 0x7f88a815e160 2023-11-24T06:19:19.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.215+0000 7f88a559a700 1 -- 172.21.15.89:0/3611168328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c0036a0 con 0x7f88a815e160 2023-11-24T06:19:19.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.216+0000 7f88aca80700 1 -- 172.21.15.89:0/3611168328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 msgr2=0x7f88a80b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.216+0000 7f88aca80700 1 --2- 172.21.15.89:0/3611168328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a80b5da0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f889c009840 tx=0x7f889c009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:19.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.216+0000 7f88aca80700 1 -- 172.21.15.89:0/3611168328 shutdown_connections 2023-11-24T06:19:19.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.216+0000 7f88aca80700 1 --2- 172.21.15.89:0/3611168328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a80b5da0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.216+0000 7f88aca80700 1 -- 172.21.15.89:0/3611168328 >> 172.21.15.89:0/3611168328 conn(0x7f88a80d0bf0 msgr2=0x7f88a80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:19.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.216+0000 7f88aca80700 1 -- 172.21.15.89:0/3611168328 shutdown_connections 2023-11-24T06:19:19.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.216+0000 7f88aca80700 1 -- 172.21.15.89:0/3611168328 wait complete. 2023-11-24T06:19:19.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.217+0000 7f88aca80700 1 Processor -- start 2023-11-24T06:19:19.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.217+0000 7f88aca80700 1 -- start start 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88aca80700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a80746a0 0x7f88a814d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88aca80700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a80afc90 0x7f88a80c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88aca80700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a81f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88aca80700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a814ae60 con 0x7f88a815e160 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88aca80700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80c9e10 con 0x7f88a80afc90 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88aca80700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80ca0c0 con 0x7f88a80746a0 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a81f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a81f3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36064/0 (socket says 172.21.15.89:36064) 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 -- 172.21.15.89:0/4057142669 learned_addr learned my addr 172.21.15.89:0/4057142669 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:19.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a5d9b700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a80afc90 0x7f88a80c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a659c700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a80746a0 0x7f88a814d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 -- 172.21.15.89:0/4057142669 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a80746a0 msgr2=0x7f88a814d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a80746a0 0x7f88a814d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 -- 172.21.15.89:0/4057142669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a80afc90 msgr2=0x7f88a80c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a80afc90 0x7f88a80c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 -- 172.21.15.89:0/4057142669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c0094a0 con 0x7f88a815e160 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88a6d9d700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a81f3330 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f88a000cba0 tx=0x7f88a000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f8896ffd700 1 -- 172.21.15.89:0/4057142669 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a000aaa0 con 0x7f88a815e160 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f8896ffd700 1 -- 172.21.15.89:0/4057142669 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a000ac00 con 0x7f88a815e160 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a814a810 con 0x7f88a815e160 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.218+0000 7f8896ffd700 1 -- 172.21.15.89:0/4057142669 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a00070d0 con 0x7f88a815e160 2023-11-24T06:19:19.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.219+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a80ca3a0 con 0x7f88a815e160 2023-11-24T06:19:19.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.219+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a80b51c0 con 0x7f88a815e160 2023-11-24T06:19:19.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.220+0000 7f8896ffd700 1 -- 172.21.15.89:0/4057142669 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f88a0007230 con 0x7f88a815e160 2023-11-24T06:19:19.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.220+0000 7f8896ffd700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f888806aed0 0x7f888806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.220+0000 7f88a659c700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f888806aed0 0x7f888806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.220+0000 7f8896ffd700 1 -- 172.21.15.89:0/4057142669 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88a00892d0 con 0x7f88a815e160 2023-11-24T06:19:19.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.221+0000 7f88a659c700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f888806aed0 0x7f888806d390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f889c0090b0 tx=0x7f889c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.225+0000 7f8896ffd700 1 -- 172.21.15.89:0/4057142669 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88a0054380 con 0x7f88a815e160 2023-11-24T06:19:19.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.414+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88a806f320 con 0x7f88a815e160 2023-11-24T06:19:19.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.415+0000 7f8896ffd700 1 -- 172.21.15.89:0/4057142669 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88a00579a0 con 0x7f88a815e160 2023-11-24T06:19:19.418 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:19.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f888806aed0 msgr2=0x7f888806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f888806aed0 0x7f888806d390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f889c0090b0 tx=0x7f889c008030 comp rx=0 tx=0).stop 2023-11-24T06:19:19.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 msgr2=0x7f88a81f3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a81f3330 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f88a000cba0 tx=0x7f88a000ceb0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 shutdown_connections 2023-11-24T06:19:19.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88a815e160 0x7f88a81f3330 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a80746a0 0x7f88a814d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f888806aed0 0x7f888806d390 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 --2- 172.21.15.89:0/4057142669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a80afc90 0x7f88a80c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 >> 172.21.15.89:0/4057142669 conn(0x7f88a80d0bf0 msgr2=0x7f88a8162690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:19.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 shutdown_connections 2023-11-24T06:19:19.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.417+0000 7f88aca80700 1 -- 172.21.15.89:0/4057142669 wait complete. 2023-11-24T06:19:19.432 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:19.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.551+0000 7f9e61698700 1 Processor -- start 2023-11-24T06:19:19.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.551+0000 7f9e61698700 1 -- start start 2023-11-24T06:19:19.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.552+0000 7f9e61698700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e5c15a620 0x7f9e5c0ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.552+0000 7f9e61698700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0b2f20 con 0x7f9e5c0cc540 2023-11-24T06:19:19.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.552+0000 7f9e61698700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0ce690 con 0x7f9e5c06ecf0 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.552+0000 7f9e61698700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c15aa00 con 0x7f9e5c15a620 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.552+0000 7f9e5affd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e5c15a620 0x7f9e5c0ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.552+0000 7f9e5a7fc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9e5c0cc540 0x7f9e5c06e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39572/0 (socket says 172.21.15.89:39572) 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.552+0000 7f9e5a7fc700 1 -- 172.21.15.89:0/1724386477 learned_addr learned my addr 172.21.15.89:0/1724386477 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e5affd700 1 -- 172.21.15.89:0/1724386477 >> v1:172.21.15.123:6789/0 conn(0x7f9e5c06ecf0 legacy=0x7f9e5c0b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e5affd700 1 -- 172.21.15.89:0/1724386477 >> v1:172.21.15.89:6789/0 conn(0x7f9e5c0cc540 legacy=0x7f9e5c06e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e59ffb700 1 -- 172.21.15.89:0/1724386477 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005172083 0 0) 0x7f9e5c0b2f20 con 0x7f9e5c0cc540 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e5affd700 1 -- 172.21.15.89:0/1724386477 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0ce960 con 0x7f9e5c15a620 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e5affd700 1 --2- 172.21.15.89:0/1724386477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c0ce150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009840 tx=0x7f9e50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4b7965ccf1aec1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e59ffb700 1 -- 172.21.15.89:0/1724386477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e50004230 con 0x7f9e5c15a620 2023-11-24T06:19:19.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e59ffb700 1 -- 172.21.15.89:0/1724386477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e50004390 con 0x7f9e5c15a620 2023-11-24T06:19:19.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e59ffb700 1 -- 172.21.15.89:0/1724386477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e500036a0 con 0x7f9e5c15a620 2023-11-24T06:19:19.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e61698700 1 -- 172.21.15.89:0/1724386477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 msgr2=0x7f9e5c0ce150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e61698700 1 --2- 172.21.15.89:0/1724386477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c0ce150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009840 tx=0x7f9e50009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:19.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e61698700 1 -- 172.21.15.89:0/1724386477 shutdown_connections 2023-11-24T06:19:19.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e61698700 1 --2- 172.21.15.89:0/1724386477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c0ce150 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.553+0000 7f9e61698700 1 -- 172.21.15.89:0/1724386477 >> 172.21.15.89:0/1724386477 conn(0x7f9e5c0b2050 msgr2=0x7f9e5c0b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:19.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.554+0000 7f9e61698700 1 -- 172.21.15.89:0/1724386477 shutdown_connections 2023-11-24T06:19:19.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.554+0000 7f9e61698700 1 -- 172.21.15.89:0/1724386477 wait complete. 2023-11-24T06:19:19.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.555+0000 7f9e61698700 1 Processor -- start 2023-11-24T06:19:19.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.555+0000 7f9e61698700 1 -- start start 2023-11-24T06:19:19.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e61698700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e5c06ecf0 0x7f9e5c1e2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e61698700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e5c0cc540 0x7f9e5c1e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e61698700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e61698700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15db30 con 0x7f9e5c15a620 2023-11-24T06:19:19.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e61698700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15c410 con 0x7f9e5c0cc540 2023-11-24T06:19:19.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e61698700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15ceb0 con 0x7f9e5c06ecf0 2023-11-24T06:19:19.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36076/0 (socket says 172.21.15.89:36076) 2023-11-24T06:19:19.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 -- 172.21.15.89:0/2166191071 learned_addr learned my addr 172.21.15.89:0/2166191071 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:19.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5a7fc700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e5c0cc540 0x7f9e5c1e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5affd700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e5c06ecf0 0x7f9e5c1e2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 -- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e5c06ecf0 msgr2=0x7f9e5c1e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e5c06ecf0 0x7f9e5c1e2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 -- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e5c0cc540 msgr2=0x7f9e5c1e3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e5c0cc540 0x7f9e5c1e3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 -- 172.21.15.89:0/2166191071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e500094a0 con 0x7f9e5c15a620 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5a7fc700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e5c0cc540 0x7f9e5c1e3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e5b7fe700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c154c20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400c7a0 tx=0x7f9e5400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e437fe700 1 -- 172.21.15.89:0/2166191071 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54003e10 con 0x7f9e5c15a620 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e437fe700 1 -- 172.21.15.89:0/2166191071 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e54005070 con 0x7f9e5c15a620 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c15d190 con 0x7f9e5c15a620 2023-11-24T06:19:19.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e437fe700 1 -- 172.21.15.89:0/2166191071 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54005420 con 0x7f9e5c15a620 2023-11-24T06:19:19.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.556+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c15d750 con 0x7f9e5c15a620 2023-11-24T06:19:19.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.557+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c068d90 con 0x7f9e5c15a620 2023-11-24T06:19:19.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.558+0000 7f9e437fe700 1 -- 172.21.15.89:0/2166191071 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9e54005580 con 0x7f9e5c15a620 2023-11-24T06:19:19.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.558+0000 7f9e437fe700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9e4806aed0 0x7f9e4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.558+0000 7f9e437fe700 1 -- 172.21.15.89:0/2166191071 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e54088d00 con 0x7f9e5c15a620 2023-11-24T06:19:19.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.558+0000 7f9e5affd700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9e4806aed0 0x7f9e4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.558+0000 7f9e5affd700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9e4806aed0 0x7f9e4806d390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9e500090b0 tx=0x7f9e50008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.563+0000 7f9e437fe700 1 -- 172.21.15.89:0/2166191071 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e54053db0 con 0x7f9e5c15a620 2023-11-24T06:19:19.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.755+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e5c1405d0 con 0x7f9e5c15a620 2023-11-24T06:19:19.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.755+0000 7f9e437fe700 1 -- 172.21.15.89:0/2166191071 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e54017070 con 0x7f9e5c15a620 2023-11-24T06:19:19.758 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:19.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9e4806aed0 msgr2=0x7f9e4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9e4806aed0 0x7f9e4806d390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9e500090b0 tx=0x7f9e50008030 comp rx=0 tx=0).stop 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 msgr2=0x7f9e5c154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c154c20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400c7a0 tx=0x7f9e5400cb60 comp rx=0 tx=0).stop 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 shutdown_connections 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c15a620 0x7f9e5c154c20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e5c06ecf0 0x7f9e5c1e2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9e4806aed0 0x7f9e4806d390 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 --2- 172.21.15.89:0/2166191071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e5c0cc540 0x7f9e5c1e3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 >> 172.21.15.89:0/2166191071 conn(0x7f9e5c0b2050 msgr2=0x7f9e5c156ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 shutdown_connections 2023-11-24T06:19:19.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.757+0000 7f9e61698700 1 -- 172.21.15.89:0/2166191071 wait complete. 2023-11-24T06:19:19.772 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_create' 2023-11-24T06:19:19.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.890+0000 7f868388d700 1 Processor -- start 2023-11-24T06:19:19.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.890+0000 7f868388d700 1 -- start start 2023-11-24T06:19:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f868388d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f867c15bdd0 0x7f867c1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f868388d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0b3370 0x7f867c0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f868388d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f867c0d1c70 0x7f867c0d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f868388d700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867c0b2f50 con 0x7f867c0d1c70 2023-11-24T06:19:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f868388d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867c0d2660 con 0x7f867c15bdd0 2023-11-24T06:19:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f868388d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0b2030 con 0x7f867c0b3370 2023-11-24T06:19:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f867c0d1c70 0x7f867c0d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f867c0d1c70 0x7f867c0d2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36084/0 (socket says 172.21.15.89:36084) 2023-11-24T06:19:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 -- 172.21.15.89:0/720841987 learned_addr learned my addr 172.21.15.89:0/720841987 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8680e28700 1 --2- 172.21.15.89:0/720841987 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0b3370 0x7f867c0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681629700 1 --2- 172.21.15.89:0/720841987 >> v2:172.21.15.123:3300/0 conn(0x7f867c15bdd0 0x7f867c1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 -- 172.21.15.89:0/720841987 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0b3370 msgr2=0x7f867c0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 --2- 172.21.15.89:0/720841987 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0b3370 0x7f867c0b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 -- 172.21.15.89:0/720841987 >> v2:172.21.15.123:3300/0 conn(0x7f867c15bdd0 msgr2=0x7f867c1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 --2- 172.21.15.89:0/720841987 >> v2:172.21.15.123:3300/0 conn(0x7f867c15bdd0 0x7f867c1604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 -- 172.21.15.89:0/720841987 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c0b22d0 con 0x7f867c0d1c70 2023-11-24T06:19:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.891+0000 7f8681e2a700 1 --2- 172.21.15.89:0/720841987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0d1c70 0x7f867c0d2120 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f867c06ec10 tx=0x7f867400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e593c560fac25b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f8673fff700 1 -- 172.21.15.89:0/720841987 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867400e070 con 0x7f867c0d1c70 2023-11-24T06:19:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f8673fff700 1 -- 172.21.15.89:0/720841987 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867400b9c0 con 0x7f867c0d1c70 2023-11-24T06:19:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f8673fff700 1 -- 172.21.15.89:0/720841987 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867400bd70 con 0x7f867c0d1c70 2023-11-24T06:19:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 -- 172.21.15.89:0/720841987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0d1c70 msgr2=0x7f867c0d2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 --2- 172.21.15.89:0/720841987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0d1c70 0x7f867c0d2120 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f867c06ec10 tx=0x7f867400b330 comp rx=0 tx=0).stop 2023-11-24T06:19:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 -- 172.21.15.89:0/720841987 shutdown_connections 2023-11-24T06:19:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 --2- 172.21.15.89:0/720841987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0d1c70 0x7f867c0d2120 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 --2- 172.21.15.89:0/720841987 >> v2:172.21.15.123:3300/0 conn(0x7f867c15bdd0 0x7f867c1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 --2- 172.21.15.89:0/720841987 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0b3370 0x7f867c0b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 -- 172.21.15.89:0/720841987 >> 172.21.15.89:0/720841987 conn(0x7f867c0731a0 msgr2=0x7f867c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:19.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 -- 172.21.15.89:0/720841987 shutdown_connections 2023-11-24T06:19:19.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.892+0000 7f868388d700 1 -- 172.21.15.89:0/720841987 wait complete. 2023-11-24T06:19:19.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.893+0000 7f868388d700 1 Processor -- start 2023-11-24T06:19:19.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f868388d700 1 -- start start 2023-11-24T06:19:19.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f868388d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0b3370 0x7f867c0cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f868388d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0d1c70 0x7f867c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f868388d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f867c15bdd0 0x7f867c0c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f868388d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0ce180 con 0x7f867c0b3370 2023-11-24T06:19:19.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f868388d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c15a8c0 con 0x7f867c15bdd0 2023-11-24T06:19:19.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f868388d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0b5590 con 0x7f867c0d1c70 2023-11-24T06:19:19.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f8681629700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0b3370 0x7f867c0cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f8681629700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0b3370 0x7f867c0cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36086/0 (socket says 172.21.15.89:36086) 2023-11-24T06:19:19.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f8681629700 1 -- 172.21.15.89:0/3375902261 learned_addr learned my addr 172.21.15.89:0/3375902261 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f8680e28700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0d1c70 0x7f867c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.894+0000 7f8681e2a700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f867c15bdd0 0x7f867c0c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8681629700 1 -- 172.21.15.89:0/3375902261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0d1c70 msgr2=0x7f867c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8681629700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0d1c70 0x7f867c15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8681629700 1 -- 172.21.15.89:0/3375902261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f867c15bdd0 msgr2=0x7f867c0c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8681629700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f867c15bdd0 0x7f867c0c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8681629700 1 -- 172.21.15.89:0/3375902261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674009b30 con 0x7f867c0b3370 2023-11-24T06:19:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8681629700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0b3370 0x7f867c0cf1c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f867800e840 tx=0x7f867800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8671ffb700 1 -- 172.21.15.89:0/3375902261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678013610 con 0x7f867c0b3370 2023-11-24T06:19:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8671ffb700 1 -- 172.21.15.89:0/3375902261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8678013770 con 0x7f867c0b3370 2023-11-24T06:19:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c0cd600 con 0x7f867c0b3370 2023-11-24T06:19:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f8671ffb700 1 -- 172.21.15.89:0/3375902261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678013b20 con 0x7f867c0b3370 2023-11-24T06:19:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.895+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c0cdbc0 con 0x7f867c0b3370 2023-11-24T06:19:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.896+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867c068a80 con 0x7f867c0b3370 2023-11-24T06:19:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.897+0000 7f8671ffb700 1 -- 172.21.15.89:0/3375902261 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8678013c80 con 0x7f867c0b3370 2023-11-24T06:19:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.897+0000 7f8671ffb700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f866006aed0 0x7f866006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.897+0000 7f8671ffb700 1 -- 172.21.15.89:0/3375902261 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8678089170 con 0x7f867c0b3370 2023-11-24T06:19:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.897+0000 7f8680e28700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f866006aed0 0x7f866006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.897+0000 7f8680e28700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f866006aed0 0x7f866006d390 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f867c070c10 tx=0x7f866c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:19.901+0000 7f8671ffb700 1 -- 172.21.15.89:0/3375902261 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86780541a0 con 0x7f867c0b3370 2023-11-24T06:19:20.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:20.176+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_create"]} v 0) v1 -- 0x7f867c067f60 con 0x7f867c0b3370 2023-11-24T06:19:21.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.173+0000 7f8671ffb700 1 -- 172.21.15.89:0/3375902261 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_create"]}]=0 v54) v1 ==== 138+0+0 (secure 0 0 0) 0x7f86780577c0 con 0x7f867c0b3370 2023-11-24T06:19:21.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f866006aed0 msgr2=0x7f866006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f866006aed0 0x7f866006d390 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f867c070c10 tx=0x7f866c008040 comp rx=0 tx=0).stop 2023-11-24T06:19:21.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0b3370 msgr2=0x7f867c0cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0b3370 0x7f867c0cf1c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f867800e840 tx=0x7f867800eb50 comp rx=0 tx=0).stop 2023-11-24T06:19:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 shutdown_connections 2023-11-24T06:19:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f866006aed0 0x7f866006d390 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f867c15bdd0 0x7f867c0c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f867c0b3370 0x7f867c0cf1c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 --2- 172.21.15.89:0/3375902261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f867c0d1c70 0x7f867c15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 >> 172.21.15.89:0/3375902261 conn(0x7f867c0731a0 msgr2=0x7f867c06f470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 shutdown_connections 2023-11-24T06:19:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.177+0000 7f868388d700 1 -- 172.21.15.89:0/3375902261 wait complete. 2023-11-24T06:19:21.195 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T06:19:21.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.320+0000 7ff0581e0700 1 Processor -- start 2023-11-24T06:19:21.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.320+0000 7ff0581e0700 1 -- start start 2023-11-24T06:19:21.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff0581e0700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff05015b370 0x7ff05015b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff0581e0700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500ae5e0 con 0x7ff0500b3c60 2023-11-24T06:19:21.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff0581e0700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05015be30 con 0x7ff0501587f0 2023-11-24T06:19:21.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff0581e0700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05015c110 con 0x7ff05015b370 2023-11-24T06:19:21.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff05577b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff05015b370 0x7ff05015b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff05577b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff05015b370 0x7ff05015b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36096/0 (socket says 172.21.15.89:36096) 2023-11-24T06:19:21.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff05677d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff0500b3c60 0x7ff0500af060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39602/0 (socket says 172.21.15.89:39602) 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff05577b700 1 -- 172.21.15.89:0/1235606679 learned_addr learned my addr 172.21.15.89:0/1235606679 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff05577b700 1 -- 172.21.15.89:0/1235606679 >> v1:172.21.15.123:6789/0 conn(0x7ff0501587f0 legacy=0x7ff05006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff05577b700 1 -- 172.21.15.89:0/1235606679 >> v1:172.21.15.89:6789/0 conn(0x7ff0500b3c60 legacy=0x7ff0500af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff054f7a700 1 -- 172.21.15.89:0/1235606679 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366654797 0 0) 0x7ff0500ae5e0 con 0x7ff0500b3c60 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.321+0000 7ff05577b700 1 -- 172.21.15.89:0/1235606679 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0500ae860 con 0x7ff05015b370 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff05577b700 1 --2- 172.21.15.89:0/1235606679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05015b370 0x7ff05015b800 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff040009920 tx=0x7ff040009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1785ef61d636b21c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff054f7a700 1 -- 172.21.15.89:0/1235606679 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040004230 con 0x7ff05015b370 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff054f7a700 1 -- 172.21.15.89:0/1235606679 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040004390 con 0x7ff05015b370 2023-11-24T06:19:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff054f7a700 1 -- 172.21.15.89:0/1235606679 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040009e70 con 0x7ff05015b370 2023-11-24T06:19:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff0581e0700 1 -- 172.21.15.89:0/1235606679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05015b370 msgr2=0x7ff05015b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff0581e0700 1 --2- 172.21.15.89:0/1235606679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05015b370 0x7ff05015b800 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff040009920 tx=0x7ff040009c30 comp rx=0 tx=0).stop 2023-11-24T06:19:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff0581e0700 1 -- 172.21.15.89:0/1235606679 shutdown_connections 2023-11-24T06:19:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff0581e0700 1 --2- 172.21.15.89:0/1235606679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05015b370 0x7ff05015b800 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff0581e0700 1 -- 172.21.15.89:0/1235606679 >> 172.21.15.89:0/1235606679 conn(0x7ff0500d0bf0 msgr2=0x7ff0500d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff0581e0700 1 -- 172.21.15.89:0/1235606679 shutdown_connections 2023-11-24T06:19:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.322+0000 7ff0581e0700 1 -- 172.21.15.89:0/1235606679 wait complete. 2023-11-24T06:19:21.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.323+0000 7ff0581e0700 1 Processor -- start 2023-11-24T06:19:21.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff0581e0700 1 -- start start 2023-11-24T06:19:21.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff0581e0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0500b3c60 0x7ff0500c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff0581e0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0501587f0 0x7ff0501f5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff0581e0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff05015b370 0x7ff05014a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff0581e0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05009d090 con 0x7ff0500b3c60 2023-11-24T06:19:21.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff0581e0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05009cd10 con 0x7ff0501587f0 2023-11-24T06:19:21.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff0581e0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0501f5780 con 0x7ff05015b370 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff055f7c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0500b3c60 0x7ff0500c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff055f7c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0500b3c60 0x7ff0500c5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36100/0 (socket says 172.21.15.89:36100) 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.324+0000 7ff055f7c700 1 -- 172.21.15.89:0/555120373 learned_addr learned my addr 172.21.15.89:0/555120373 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff05577b700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0501587f0 0x7ff0501f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff05677d700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff05015b370 0x7ff05014a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff055f7c700 1 -- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff05015b370 msgr2=0x7ff05014a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff055f7c700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff05015b370 0x7ff05014a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff055f7c700 1 -- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0501587f0 msgr2=0x7ff0501f5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff055f7c700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0501587f0 0x7ff0501f5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff055f7c700 1 -- 172.21.15.89:0/555120373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff040009550 con 0x7ff0500b3c60 2023-11-24T06:19:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff05577b700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0501587f0 0x7ff0501f5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff05677d700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff05015b370 0x7ff05014a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff055f7c700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0500b3c60 0x7ff0500c5d20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c4d0 tx=0x7ff04c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff0467fc700 1 -- 172.21.15.89:0/555120373 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c011790 con 0x7ff0500b3c60 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff0467fc700 1 -- 172.21.15.89:0/555120373 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04c0118f0 con 0x7ff0500b3c60 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff0467fc700 1 -- 172.21.15.89:0/555120373 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c011ca0 con 0x7ff0500b3c60 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0501f5a20 con 0x7ff0500b3c60 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.325+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0501f5fe0 con 0x7ff0500b3c60 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.326+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff050068fb0 con 0x7ff0500b3c60 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.326+0000 7ff0467fc700 1 -- 172.21.15.89:0/555120373 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff04c01c460 con 0x7ff0500b3c60 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.327+0000 7ff0467fc700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff03406aed0 0x7ff03406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.327+0000 7ff0467fc700 1 -- 172.21.15.89:0/555120373 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff04c0893d0 con 0x7ff0500b3c60 2023-11-24T06:19:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.327+0000 7ff05577b700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff03406aed0 0x7ff03406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.327+0000 7ff05577b700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff03406aed0 0x7ff03406d390 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff05015abe0 tx=0x7ff040008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.331+0000 7ff0467fc700 1 -- 172.21.15.89:0/555120373 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff04c054510 con 0x7ff0500b3c60 2023-11-24T06:19:21.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.522+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff0500b24d0 con 0x7ff0500b3c60 2023-11-24T06:19:21.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.525+0000 7ff0467fc700 1 -- 172.21.15.89:0/555120373 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff04c057b30 con 0x7ff0500b3c60 2023-11-24T06:19:21.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff03406aed0 msgr2=0x7ff03406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff03406aed0 0x7ff03406d390 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff05015abe0 tx=0x7ff040008030 comp rx=0 tx=0).stop 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0500b3c60 msgr2=0x7ff0500c5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0500b3c60 0x7ff0500c5d20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c4d0 tx=0x7ff04c00c890 comp rx=0 tx=0).stop 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 shutdown_connections 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff05015b370 0x7ff05014a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0500b3c60 0x7ff0500c5d20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff03406aed0 0x7ff03406d390 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 --2- 172.21.15.89:0/555120373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0501587f0 0x7ff0501f5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 >> 172.21.15.89:0/555120373 conn(0x7ff0500d0bf0 msgr2=0x7ff05006f250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 shutdown_connections 2023-11-24T06:19:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.527+0000 7ff0581e0700 1 -- 172.21.15.89:0/555120373 wait complete. 2023-11-24T06:19:21.543 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T06:19:21.543 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:19:21.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:19:21.630 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T06:19:21.631 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:21.631 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:21.648 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:21.648 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:21.706 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:21.706 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:21.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:21.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.920+0000 7f5284a85700 1 Processor -- start 2023-11-24T06:19:21.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.920+0000 7f5284a85700 1 -- start start 2023-11-24T06:19:21.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f5284a85700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f528015e080 0x7f52800b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f5284a85700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52800af4d0 0x7f52800afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f5284a85700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f5284a85700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52800af0b0 con 0x7f52800af4d0 2023-11-24T06:19:21.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f5284a85700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528006c9f0 con 0x7f528015e080 2023-11-24T06:19:21.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f5284a85700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52800b05a0 con 0x7f528006c000 2023-11-24T06:19:21.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f527dd9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52800af4d0 0x7f52800afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f527dd9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52800af4d0 0x7f52800afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36108/0 (socket says 172.21.15.89:36108) 2023-11-24T06:19:21.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.921+0000 7f527dd9b700 1 -- 172.21.15.89:0/3783528401 learned_addr learned my addr 172.21.15.89:0/3783528401 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:21.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527ed9d700 1 --2- 172.21.15.89:0/3783528401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527e59c700 1 --2- 172.21.15.89:0/3783528401 >> v2:172.21.15.123:3300/0 conn(0x7f528015e080 0x7f52800b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527dd9b700 1 -- 172.21.15.89:0/3783528401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 msgr2=0x7f528006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:21.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527dd9b700 1 --2- 172.21.15.89:0/3783528401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528006c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527dd9b700 1 -- 172.21.15.89:0/3783528401 >> v2:172.21.15.123:3300/0 conn(0x7f528015e080 msgr2=0x7f52800b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:21.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527dd9b700 1 --2- 172.21.15.89:0/3783528401 >> v2:172.21.15.123:3300/0 conn(0x7f528015e080 0x7f52800b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527dd9b700 1 -- 172.21.15.89:0/3783528401 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52800ae430 con 0x7f52800af4d0 2023-11-24T06:19:21.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527ed9d700 1 --2- 172.21.15.89:0/3783528401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:21.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527e59c700 1 --2- 172.21.15.89:0/3783528401 >> v2:172.21.15.123:3300/0 conn(0x7f528015e080 0x7f52800b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:21.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527dd9b700 1 --2- 172.21.15.89:0/3783528401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52800af4d0 0x7f52800afb90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5280072a90 tx=0x7f5270009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e72fab8c38995c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:21.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527d59a700 1 -- 172.21.15.89:0/3783528401 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527000b870 con 0x7f52800af4d0 2023-11-24T06:19:21.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527d59a700 1 -- 172.21.15.89:0/3783528401 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527000b9d0 con 0x7f52800af4d0 2023-11-24T06:19:21.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f527d59a700 1 -- 172.21.15.89:0/3783528401 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527000bd60 con 0x7f52800af4d0 2023-11-24T06:19:21.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f5284a85700 1 -- 172.21.15.89:0/3783528401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52800af4d0 msgr2=0x7f52800afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.922+0000 7f5284a85700 1 --2- 172.21.15.89:0/3783528401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52800af4d0 0x7f52800afb90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5280072a90 tx=0x7f5270009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.923+0000 7f5284a85700 1 -- 172.21.15.89:0/3783528401 shutdown_connections 2023-11-24T06:19:21.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.923+0000 7f5284a85700 1 --2- 172.21.15.89:0/3783528401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.923+0000 7f5284a85700 1 --2- 172.21.15.89:0/3783528401 >> v2:172.21.15.123:3300/0 conn(0x7f528015e080 0x7f52800b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.923+0000 7f5284a85700 1 --2- 172.21.15.89:0/3783528401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52800af4d0 0x7f52800afb90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.923+0000 7f5284a85700 1 -- 172.21.15.89:0/3783528401 >> 172.21.15.89:0/3783528401 conn(0x7f5280071b90 msgr2=0x7f5280071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:21.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.923+0000 7f5284a85700 1 -- 172.21.15.89:0/3783528401 shutdown_connections 2023-11-24T06:19:21.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.923+0000 7f5284a85700 1 -- 172.21.15.89:0/3783528401 wait complete. 2023-11-24T06:19:21.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.924+0000 7f5284a85700 1 Processor -- start 2023-11-24T06:19:21.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.924+0000 7f5284a85700 1 -- start start 2023-11-24T06:19:21.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f5284a85700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528014c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f5284a85700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52800af4d0 0x7f52801f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f5284a85700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f528015e080 0x7f52801605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f5284a85700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5280158c60 con 0x7f528015e080 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f5284a85700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5280158050 con 0x7f52800af4d0 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f5284a85700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5280158330 con 0x7f528006c000 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f528015e080 0x7f52801605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f528015e080 0x7f52801605c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36116/0 (socket says 172.21.15.89:36116) 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 -- 172.21.15.89:0/2452171484 learned_addr learned my addr 172.21.15.89:0/2452171484 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527e59c700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528014c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527dd9b700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52800af4d0 0x7f52801f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 -- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 msgr2=0x7f528014c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528014c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 -- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52800af4d0 msgr2=0x7f52801f36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52800af4d0 0x7f52801f36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 -- 172.21.15.89:0/2452171484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5270009550 con 0x7f528015e080 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527e59c700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528014c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f527ed9d700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f528015e080 0x7f52801605c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f527800cef0 tx=0x7f527800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f526effd700 1 -- 172.21.15.89:0/2452171484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527800add0 con 0x7f528015e080 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f526effd700 1 -- 172.21.15.89:0/2452171484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278015440 con 0x7f528015e080 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5280158610 con 0x7f528015e080 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f526effd700 1 -- 172.21.15.89:0/2452171484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52780157f0 con 0x7f528015e080 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.925+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5280157c70 con 0x7f528015e080 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.926+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52800753f0 con 0x7f528015e080 2023-11-24T06:19:21.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.927+0000 7f526effd700 1 -- 172.21.15.89:0/2452171484 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f527800d430 con 0x7f528015e080 2023-11-24T06:19:21.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.927+0000 7f526effd700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f526006aed0 0x7f526006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:21.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.927+0000 7f526effd700 1 -- 172.21.15.89:0/2452171484 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5278089ba0 con 0x7f528015e080 2023-11-24T06:19:21.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.927+0000 7f527e59c700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f526006aed0 0x7f526006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:21.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.927+0000 7f527e59c700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f526006aed0 0x7f526006d390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5274008910 tx=0x7f5274008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:21.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:21.931+0000 7f526effd700 1 -- 172.21.15.89:0/2452171484 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5278054c50 con 0x7f528015e080 2023-11-24T06:19:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.127+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52800b0080 con 0x7f528015e080 2023-11-24T06:19:22.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.128+0000 7f526effd700 1 -- 172.21.15.89:0/2452171484 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f527801a020 con 0x7f528015e080 2023-11-24T06:19:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:22.131 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:22.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f526006aed0 msgr2=0x7f526006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f526006aed0 0x7f526006d390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5274008910 tx=0x7f5274008040 comp rx=0 tx=0).stop 2023-11-24T06:19:22.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f528015e080 msgr2=0x7f52801605c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f528015e080 0x7f52801605c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f527800cef0 tx=0x7f527800a340 comp rx=0 tx=0).stop 2023-11-24T06:19:22.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 shutdown_connections 2023-11-24T06:19:22.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f528015e080 0x7f52801605c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f528006c000 0x7f528014c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f526006aed0 0x7f526006d390 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 --2- 172.21.15.89:0/2452171484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52800af4d0 0x7f52801f36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 >> 172.21.15.89:0/2452171484 conn(0x7f5280071b90 msgr2=0x7f52800b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:22.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 shutdown_connections 2023-11-24T06:19:22.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.130+0000 7f5284a85700 1 -- 172.21.15.89:0/2452171484 wait complete. 2023-11-24T06:19:22.145 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:22.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:22.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.268+0000 7f2ab866d700 1 Processor -- start 2023-11-24T06:19:22.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.268+0000 7f2ab866d700 1 -- start start 2023-11-24T06:19:22.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.268+0000 7f2ab866d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2ab015a620 0x7f2ab00ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab866d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2ab006ecf0 0x7f2ab00b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab866d700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab00b2f20 con 0x7f2ab00cc540 2023-11-24T06:19:22.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab866d700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab00ce690 con 0x7f2ab006ecf0 2023-11-24T06:19:22.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab866d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab015aa00 con 0x7f2ab015a620 2023-11-24T06:19:22.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6c0a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2ab006ecf0 0x7f2ab00b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6c0a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2ab006ecf0 0x7f2ab00b3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36122/0 (socket says 172.21.15.89:36122) 2023-11-24T06:19:22.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6c0a700 1 -- 172.21.15.89:0/2860976931 learned_addr learned my addr 172.21.15.89:0/2860976931 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:22.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6409700 1 --2- 172.21.15.89:0/2860976931 >> v2:172.21.15.123:3300/0 conn(0x7f2ab015a620 0x7f2ab00ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6c0a700 1 -- 172.21.15.89:0/2860976931 >> v2:172.21.15.123:3300/0 conn(0x7f2ab015a620 msgr2=0x7f2ab00ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:22.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6c0a700 1 --2- 172.21.15.89:0/2860976931 >> v2:172.21.15.123:3300/0 conn(0x7f2ab015a620 0x7f2ab00ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6c0a700 1 -- 172.21.15.89:0/2860976931 >> v1:172.21.15.123:6789/0 conn(0x7f2ab00cc540 legacy=0x7f2ab006e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6c0a700 1 -- 172.21.15.89:0/2860976931 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab00ce960 con 0x7f2ab006ecf0 2023-11-24T06:19:22.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab6c0a700 1 --2- 172.21.15.89:0/2860976931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab006ecf0 0x7f2ab00b3350 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0072800 tx=0x7f2aa800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d134d125804e0c6c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab5407700 1 -- 172.21.15.89:0/2860976931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800e070 con 0x7f2ab006ecf0 2023-11-24T06:19:22.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab5407700 1 -- 172.21.15.89:0/2860976931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa800b9c0 con 0x7f2ab006ecf0 2023-11-24T06:19:22.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.269+0000 7f2ab5407700 1 -- 172.21.15.89:0/2860976931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800bd70 con 0x7f2ab006ecf0 2023-11-24T06:19:22.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.270+0000 7f2ab866d700 1 -- 172.21.15.89:0/2860976931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab006ecf0 msgr2=0x7f2ab00b3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.270+0000 7f2ab866d700 1 --2- 172.21.15.89:0/2860976931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab006ecf0 0x7f2ab00b3350 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0072800 tx=0x7f2aa800b330 comp rx=0 tx=0).stop 2023-11-24T06:19:22.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.270+0000 7f2ab866d700 1 -- 172.21.15.89:0/2860976931 shutdown_connections 2023-11-24T06:19:22.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.270+0000 7f2ab866d700 1 --2- 172.21.15.89:0/2860976931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab006ecf0 0x7f2ab00b3350 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.270+0000 7f2ab866d700 1 --2- 172.21.15.89:0/2860976931 >> v2:172.21.15.123:3300/0 conn(0x7f2ab015a620 0x7f2ab00ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.270+0000 7f2ab866d700 1 -- 172.21.15.89:0/2860976931 >> 172.21.15.89:0/2860976931 conn(0x7f2ab00b2050 msgr2=0x7f2ab00b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:22.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.270+0000 7f2ab866d700 1 -- 172.21.15.89:0/2860976931 shutdown_connections 2023-11-24T06:19:22.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.270+0000 7f2ab866d700 1 -- 172.21.15.89:0/2860976931 wait complete. 2023-11-24T06:19:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.271+0000 7f2ab866d700 1 Processor -- start 2023-11-24T06:19:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.271+0000 7f2ab866d700 1 -- start start 2023-11-24T06:19:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab866d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ab006ecf0 0x7f2ab01451f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab866d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab00cc540 0x7f2ab014f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab866d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab015a620 0x7f2ab014fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab866d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab0150350 con 0x7f2ab00cc540 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab866d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab0131c40 con 0x7f2ab015a620 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab866d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab0150070 con 0x7f2ab006ecf0 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab5c08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab00cc540 0x7f2ab014f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab5c08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab00cc540 0x7f2ab014f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36126/0 (socket says 172.21.15.89:36126) 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab5c08700 1 -- 172.21.15.89:0/1446434359 learned_addr learned my addr 172.21.15.89:0/1446434359 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab6c0a700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab015a620 0x7f2ab014fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab6409700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ab006ecf0 0x7f2ab01451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab5c08700 1 -- 172.21.15.89:0/1446434359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ab006ecf0 msgr2=0x7f2ab01451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab5c08700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ab006ecf0 0x7f2ab01451f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab5c08700 1 -- 172.21.15.89:0/1446434359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab015a620 msgr2=0x7f2ab014fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab5c08700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab015a620 0x7f2ab014fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.272+0000 7f2ab5c08700 1 -- 172.21.15.89:0/1446434359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8009b30 con 0x7f2ab00cc540 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.273+0000 7f2ab5c08700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab00cc540 0x7f2ab014f4d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c510 tx=0x7f2aac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.273+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1446434359 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac0052d0 con 0x7f2ab00cc540 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.273+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1446434359 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac005450 con 0x7f2ab00cc540 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.273+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab01315f0 con 0x7f2ab00cc540 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.273+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1446434359 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac0038c0 con 0x7f2ab00cc540 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.273+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab0145cb0 con 0x7f2ab00cc540 2023-11-24T06:19:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.274+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab0068d90 con 0x7f2ab00cc540 2023-11-24T06:19:22.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.274+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1446434359 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2aac003a20 con 0x7f2ab00cc540 2023-11-24T06:19:22.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.274+0000 7f2aa6ffd700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a9406aed0 0x7f2a9406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.275+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1446434359 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2aac088970 con 0x7f2ab00cc540 2023-11-24T06:19:22.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.275+0000 7f2ab6409700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a9406aed0 0x7f2a9406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.275+0000 7f2ab6409700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a9406aed0 0x7f2a9406d390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0159b90 tx=0x7f2aa0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.279+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1446434359 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2aac053ab0 con 0x7f2ab00cc540 2023-11-24T06:19:22.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.467+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ab0131450 con 0x7f2ab00cc540 2023-11-24T06:19:22.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.467+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1446434359 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ab0131450 con 0x7f2ab00cc540 2023-11-24T06:19:22.471 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:22.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.469+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a9406aed0 msgr2=0x7f2a9406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.469+0000 7f2ab866d700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a9406aed0 0x7f2a9406d390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0159b90 tx=0x7f2aa0008040 comp rx=0 tx=0).stop 2023-11-24T06:19:22.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.469+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab00cc540 msgr2=0x7f2ab014f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.469+0000 7f2ab866d700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab00cc540 0x7f2ab014f4d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c510 tx=0x7f2aac00c8d0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.470+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 shutdown_connections 2023-11-24T06:19:22.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.470+0000 7f2ab866d700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2a9406aed0 0x7f2a9406d390 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.470+0000 7f2ab866d700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab015a620 0x7f2ab014fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.470+0000 7f2ab866d700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ab006ecf0 0x7f2ab01451f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.470+0000 7f2ab866d700 1 --2- 172.21.15.89:0/1446434359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab00cc540 0x7f2ab014f4d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.470+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 >> 172.21.15.89:0/1446434359 conn(0x7f2ab00b2050 msgr2=0x7f2ab006e570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:22.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.470+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 shutdown_connections 2023-11-24T06:19:22.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.470+0000 7f2ab866d700 1 -- 172.21.15.89:0/1446434359 wait complete. 2023-11-24T06:19:22.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:22.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.604+0000 7fbb189a1700 1 Processor -- start 2023-11-24T06:19:22.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.604+0000 7fbb189a1700 1 -- start start 2023-11-24T06:19:22.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.605+0000 7fbb189a1700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbb1415e160 0x7fbb140b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.605+0000 7fbb189a1700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbb140afc90 0x7fbb14073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.605+0000 7fbb189a1700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb140b3c60 con 0x7fbb140744d0 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.605+0000 7fbb189a1700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb140b4500 con 0x7fbb1415e160 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.605+0000 7fbb189a1700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb14074940 con 0x7fbb140afc90 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.605+0000 7fbb1259c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbb1415e160 0x7fbb140b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.605+0000 7fbb12d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbb140744d0 0x7fbb1406d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39640/0 (socket says 172.21.15.89:39640) 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.605+0000 7fbb12d9d700 1 -- 172.21.15.89:0/1679512447 learned_addr learned my addr 172.21.15.89:0/1679512447 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb11d9b700 1 --2- 172.21.15.89:0/1679512447 >> v2:172.21.15.123:3300/0 conn(0x7fbb140afc90 0x7fbb14073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1259c700 1 -- 172.21.15.89:0/1679512447 >> v2:172.21.15.123:3300/0 conn(0x7fbb140afc90 msgr2=0x7fbb14073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1259c700 1 --2- 172.21.15.89:0/1679512447 >> v2:172.21.15.123:3300/0 conn(0x7fbb140afc90 0x7fbb14073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1259c700 1 -- 172.21.15.89:0/1679512447 >> v1:172.21.15.89:6789/0 conn(0x7fbb140744d0 legacy=0x7fbb1406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1259c700 1 -- 172.21.15.89:0/1679512447 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb140b58c0 con 0x7fbb1415e160 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1159a700 1 -- 172.21.15.89:0/1679512447 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731272490 0 0) 0x7fbb140b3c60 con 0x7fbb140744d0 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb11d9b700 1 --2- 172.21.15.89:0/1679512447 >> v2:172.21.15.123:3300/0 conn(0x7fbb140afc90 0x7fbb14073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1259c700 1 --2- 172.21.15.89:0/1679512447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb140b5da0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbb140d0880 tx=0x7fbb08009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e0dd000c194b62 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1159a700 1 -- 172.21.15.89:0/1679512447 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08011830 con 0x7fbb1415e160 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1159a700 1 -- 172.21.15.89:0/1679512447 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb08011990 con 0x7fbb1415e160 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.606+0000 7fbb1159a700 1 -- 172.21.15.89:0/1679512447 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08011d20 con 0x7fbb1415e160 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.607+0000 7fbb189a1700 1 -- 172.21.15.89:0/1679512447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 msgr2=0x7fbb140b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.607+0000 7fbb189a1700 1 --2- 172.21.15.89:0/1679512447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb140b5da0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbb140d0880 tx=0x7fbb08009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.607+0000 7fbb189a1700 1 -- 172.21.15.89:0/1679512447 shutdown_connections 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.607+0000 7fbb189a1700 1 --2- 172.21.15.89:0/1679512447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb140b5da0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.607+0000 7fbb189a1700 1 --2- 172.21.15.89:0/1679512447 >> v2:172.21.15.123:3300/0 conn(0x7fbb140afc90 0x7fbb14073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.607+0000 7fbb189a1700 1 -- 172.21.15.89:0/1679512447 >> 172.21.15.89:0/1679512447 conn(0x7fbb140d0bf0 msgr2=0x7fbb140d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:22.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.607+0000 7fbb189a1700 1 -- 172.21.15.89:0/1679512447 shutdown_connections 2023-11-24T06:19:22.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.607+0000 7fbb189a1700 1 -- 172.21.15.89:0/1679512447 wait complete. 2023-11-24T06:19:22.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.608+0000 7fbb189a1700 1 Processor -- start 2023-11-24T06:19:22.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.608+0000 7fbb189a1700 1 -- start start 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb189a1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb140744d0 0x7fbb1414a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb189a1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140afc90 0x7fbb140adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb189a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb1414d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb189a1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb14150460 con 0x7fbb1415e160 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb189a1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1414ed50 con 0x7fbb140afc90 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb189a1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1414e140 con 0x7fbb140744d0 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb1414d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb1414d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36142/0 (socket says 172.21.15.89:36142) 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 -- 172.21.15.89:0/2817334994 learned_addr learned my addr 172.21.15.89:0/2817334994 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb1259c700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb140744d0 0x7fbb1414a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 -- 172.21.15.89:0/2817334994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb140744d0 msgr2=0x7fbb1414a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb140744d0 0x7fbb1414a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb11d9b700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140afc90 0x7fbb140adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 -- 172.21.15.89:0/2817334994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140afc90 msgr2=0x7fbb140adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140afc90 0x7fbb140adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 -- 172.21.15.89:0/2817334994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb080094a0 con 0x7fbb1415e160 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb12d9d700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb1414d560 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00cba0 tx=0x7fbb0c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.609+0000 7fbb02ffd700 1 -- 172.21.15.89:0/2817334994 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00ab80 con 0x7fbb1415e160 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.610+0000 7fbb02ffd700 1 -- 172.21.15.89:0/2817334994 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c00ad00 con 0x7fbb1415e160 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.610+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1414e420 con 0x7fbb1415e160 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.610+0000 7fbb02ffd700 1 -- 172.21.15.89:0/2817334994 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c007310 con 0x7fbb1415e160 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.610+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1414e9e0 con 0x7fbb1415e160 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.610+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1414ef00 con 0x7fbb1415e160 2023-11-24T06:19:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.611+0000 7fbb02ffd700 1 -- 172.21.15.89:0/2817334994 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbb0c007470 con 0x7fbb1415e160 2023-11-24T06:19:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.611+0000 7fbb02ffd700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbaf406aed0 0x7fbaf406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.611+0000 7fbb1259c700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbaf406aed0 0x7fbaf406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.611+0000 7fbb02ffd700 1 -- 172.21.15.89:0/2817334994 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb0c0894b0 con 0x7fbb1415e160 2023-11-24T06:19:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.612+0000 7fbb1259c700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbaf406aed0 0x7fbaf406d390 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbb080097c0 tx=0x7fbb08008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.616+0000 7fbb02ffd700 1 -- 172.21.15.89:0/2817334994 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb1414ef00 con 0x7fbb1415e160 2023-11-24T06:19:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.809+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbb1414eec0 con 0x7fbb1415e160 2023-11-24T06:19:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.809+0000 7fbb02ffd700 1 -- 172.21.15.89:0/2817334994 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbb0c0544c0 con 0x7fbb1415e160 2023-11-24T06:19:22.812 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbaf406aed0 msgr2=0x7fbaf406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbaf406aed0 0x7fbaf406d390 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbb080097c0 tx=0x7fbb08008030 comp rx=0 tx=0).stop 2023-11-24T06:19:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 msgr2=0x7fbb1414d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb1414d560 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00cba0 tx=0x7fbb0c00cf60 comp rx=0 tx=0).stop 2023-11-24T06:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 shutdown_connections 2023-11-24T06:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb1415e160 0x7fbb1414d560 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb140744d0 0x7fbb1414a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbaf406aed0 0x7fbaf406d390 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 --2- 172.21.15.89:0/2817334994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140afc90 0x7fbb140adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 >> 172.21.15.89:0/2817334994 conn(0x7fbb140d0bf0 msgr2=0x7fbb1406d930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 shutdown_connections 2023-11-24T06:19:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.811+0000 7fbb189a1700 1 -- 172.21.15.89:0/2817334994 wait complete. 2023-11-24T06:19:22.826 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:22.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.946+0000 7f3122275700 1 Processor -- start 2023-11-24T06:19:22.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.946+0000 7f3122275700 1 -- start start 2023-11-24T06:19:22.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f3122275700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f311c1587f0 0x7f311c06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f3122275700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c15b370 0x7f311c15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f3122275700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c06b530 con 0x7f311c0b3c60 2023-11-24T06:19:22.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f3122275700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0ae5e0 con 0x7f311c1587f0 2023-11-24T06:19:22.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f3122275700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c15be30 con 0x7f311c15b370 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311bfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f311c1587f0 0x7f311c06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311bfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f311c1587f0 0x7f311c06aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36146/0 (socket says 172.21.15.89:36146) 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311bfff700 1 -- 172.21.15.89:0/347113481 learned_addr learned my addr 172.21.15.89:0/347113481 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311b7fe700 1 --2- 172.21.15.89:0/347113481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c15b370 0x7f311c15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311bfff700 1 -- 172.21.15.89:0/347113481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c15b370 msgr2=0x7f311c15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311bfff700 1 --2- 172.21.15.89:0/347113481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c15b370 0x7f311c15b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311bfff700 1 -- 172.21.15.89:0/347113481 >> v1:172.21.15.123:6789/0 conn(0x7f311c0b3c60 legacy=0x7f311c0af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311bfff700 1 -- 172.21.15.89:0/347113481 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c15c110 con 0x7f311c1587f0 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311bfff700 1 --2- 172.21.15.89:0/347113481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c06aff0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f31100098f0 tx=0x7f3110009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25b4b5ce5f77616 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311affd700 1 -- 172.21.15.89:0/347113481 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110011830 con 0x7f311c1587f0 2023-11-24T06:19:22.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311affd700 1 -- 172.21.15.89:0/347113481 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110011990 con 0x7f311c1587f0 2023-11-24T06:19:22.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.947+0000 7f311affd700 1 -- 172.21.15.89:0/347113481 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110011d20 con 0x7f311c1587f0 2023-11-24T06:19:22.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.948+0000 7f3122275700 1 -- 172.21.15.89:0/347113481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 msgr2=0x7f311c06aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.948+0000 7f3122275700 1 --2- 172.21.15.89:0/347113481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c06aff0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f31100098f0 tx=0x7f3110009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:22.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.948+0000 7f3122275700 1 -- 172.21.15.89:0/347113481 shutdown_connections 2023-11-24T06:19:22.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.948+0000 7f3122275700 1 --2- 172.21.15.89:0/347113481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c06aff0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.948+0000 7f3122275700 1 --2- 172.21.15.89:0/347113481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c15b370 0x7f311c15b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.948+0000 7f3122275700 1 -- 172.21.15.89:0/347113481 >> 172.21.15.89:0/347113481 conn(0x7f311c0d0bf0 msgr2=0x7f311c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.948+0000 7f3122275700 1 -- 172.21.15.89:0/347113481 shutdown_connections 2023-11-24T06:19:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.948+0000 7f3122275700 1 -- 172.21.15.89:0/347113481 wait complete. 2023-11-24T06:19:22.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.949+0000 7f3122275700 1 Processor -- start 2023-11-24T06:19:22.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.950+0000 7f3122275700 1 -- start start 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.950+0000 7f3122275700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c0b3c60 0x7f311c1481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.950+0000 7f3122275700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3122275700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f311c15b370 0x7f311c150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3122275700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c14ae50 con 0x7f311c1587f0 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3122275700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c148af0 con 0x7f311c15b370 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3122275700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c150870 con 0x7f311c0b3c60 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c0b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36154/0 (socket says 172.21.15.89:36154) 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 -- 172.21.15.89:0/1774627565 learned_addr learned my addr 172.21.15.89:0/1774627565 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:22.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311bfff700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c0b3c60 0x7f311c1481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3120812700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f311c15b370 0x7f311c150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 -- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c0b3c60 msgr2=0x7f311c1481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c0b3c60 0x7f311c1481c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 -- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f311c15b370 msgr2=0x7f311c150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f311c15b370 0x7f311c150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 -- 172.21.15.89:0/1774627565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110009550 con 0x7f311c1587f0 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311bfff700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c0b3c60 0x7f311c1481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f311b7fe700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c0b15b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e870 tx=0x7f310c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3118ff9700 1 -- 172.21.15.89:0/1774627565 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c009590 con 0x7f311c1587f0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3118ff9700 1 -- 172.21.15.89:0/1774627565 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c00ee70 con 0x7f311c1587f0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c148810 con 0x7f311c1587f0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3118ff9700 1 -- 172.21.15.89:0/1774627565 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c00ca50 con 0x7f311c1587f0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.951+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c14b360 con 0x7f311c1587f0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.952+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c14b9b0 con 0x7f311c1587f0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.953+0000 7f3118ff9700 1 -- 172.21.15.89:0/1774627565 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f310c00cbb0 con 0x7f311c1587f0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.953+0000 7f3118ff9700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f310806aed0 0x7f310806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.953+0000 7f3118ff9700 1 -- 172.21.15.89:0/1774627565 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f310c0895b0 con 0x7f311c1587f0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.953+0000 7f311bfff700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f310806aed0 0x7f310806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.953+0000 7f311bfff700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f310806aed0 0x7f310806d390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3110009c40 tx=0x7f3110008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:22.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:22.957+0000 7f3118ff9700 1 -- 172.21.15.89:0/1774627565 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f310c054660 con 0x7f311c1587f0 2023-11-24T06:19:23.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.149+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f311c068fb0 con 0x7f311c1587f0 2023-11-24T06:19:23.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.149+0000 7f3118ff9700 1 -- 172.21.15.89:0/1774627565 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f310c00ce60 con 0x7f311c1587f0 2023-11-24T06:19:23.152 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f310806aed0 msgr2=0x7f310806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f310806aed0 0x7f310806d390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3110009c40 tx=0x7f3110008030 comp rx=0 tx=0).stop 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 msgr2=0x7f311c0b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c0b15b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e870 tx=0x7f310c00ec30 comp rx=0 tx=0).stop 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 shutdown_connections 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f310806aed0 0x7f310806d390 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f311c15b370 0x7f311c150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f311c0b3c60 0x7f311c1481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 --2- 172.21.15.89:0/1774627565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f311c1587f0 0x7f311c0b15b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 >> 172.21.15.89:0/1774627565 conn(0x7f311c0d0bf0 msgr2=0x7f311c151000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:23.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 shutdown_connections 2023-11-24T06:19:23.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.151+0000 7f3122275700 1 -- 172.21.15.89:0/1774627565 wait complete. 2023-11-24T06:19:23.166 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:19:23.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.285+0000 7f10e308a700 1 Processor -- start 2023-11-24T06:19:23.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.285+0000 7f10e308a700 1 -- start start 2023-11-24T06:19:23.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e308a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10dc15bed0 0x7f10dc0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e308a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc1603c0 0x7f10dc0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e308a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10dc0d0770 con 0x7f10dc0d1a50 2023-11-24T06:19:23.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e308a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc15c2b0 con 0x7f10dc15bed0 2023-11-24T06:19:23.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e308a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0cd100 con 0x7f10dc1603c0 2023-11-24T06:19:23.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e0e26700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10dc15bed0 0x7f10dc0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e0e26700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10dc15bed0 0x7f10dc0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36160/0 (socket says 172.21.15.89:36160) 2023-11-24T06:19:23.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e0e26700 1 -- 172.21.15.89:0/1019661980 learned_addr learned my addr 172.21.15.89:0/1019661980 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:23.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10dbfff700 1 --2- 172.21.15.89:0/1019661980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc1603c0 0x7f10dc0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e0e26700 1 -- 172.21.15.89:0/1019661980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc1603c0 msgr2=0x7f10dc0d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:23.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e0e26700 1 --2- 172.21.15.89:0/1019661980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc1603c0 0x7f10dc0d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e0e26700 1 -- 172.21.15.89:0/1019661980 >> v1:172.21.15.123:6789/0 conn(0x7f10dc0d1a50 legacy=0x7f10dc0d1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:23.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e0e26700 1 -- 172.21.15.89:0/1019661980 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc0d09a0 con 0x7f10dc15bed0 2023-11-24T06:19:23.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10dbfff700 1 --2- 172.21.15.89:0/1019661980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc1603c0 0x7f10dc0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:23.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10e0e26700 1 --2- 172.21.15.89:0/1019661980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc15bed0 0x7f10dc0b19f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f10d00098f0 tx=0x7f10d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e675b64eda5dd81 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:23.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.286+0000 7f10db7fe700 1 -- 172.21.15.89:0/1019661980 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d0004230 con 0x7f10dc15bed0 2023-11-24T06:19:23.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10db7fe700 1 -- 172.21.15.89:0/1019661980 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d0004390 con 0x7f10dc15bed0 2023-11-24T06:19:23.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10db7fe700 1 -- 172.21.15.89:0/1019661980 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d00036a0 con 0x7f10dc15bed0 2023-11-24T06:19:23.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10e308a700 1 -- 172.21.15.89:0/1019661980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc15bed0 msgr2=0x7f10dc0b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10e308a700 1 --2- 172.21.15.89:0/1019661980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc15bed0 0x7f10dc0b19f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f10d00098f0 tx=0x7f10d0009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:23.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10e308a700 1 -- 172.21.15.89:0/1019661980 shutdown_connections 2023-11-24T06:19:23.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10e308a700 1 --2- 172.21.15.89:0/1019661980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc15bed0 0x7f10dc0b19f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10e308a700 1 --2- 172.21.15.89:0/1019661980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc1603c0 0x7f10dc0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10e308a700 1 -- 172.21.15.89:0/1019661980 >> 172.21.15.89:0/1019661980 conn(0x7f10dc073130 msgr2=0x7f10dc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:23.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10e308a700 1 -- 172.21.15.89:0/1019661980 shutdown_connections 2023-11-24T06:19:23.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.287+0000 7f10e308a700 1 -- 172.21.15.89:0/1019661980 wait complete. 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.288+0000 7f10e308a700 1 Processor -- start 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.288+0000 7f10e308a700 1 -- start start 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e308a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc0d1a50 0x7f10dc14b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e308a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc15bed0 0x7f10dc06fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e308a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc1603c0 0x7f10dc070060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e308a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc1f5bb0 con 0x7f10dc1603c0 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e308a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc1f25a0 con 0x7f10dc0d1a50 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e308a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc1f17e0 con 0x7f10dc15bed0 2023-11-24T06:19:23.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e1627700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc1603c0 0x7f10dc070060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e1627700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc1603c0 0x7f10dc070060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36170/0 (socket says 172.21.15.89:36170) 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e1627700 1 -- 172.21.15.89:0/894888409 learned_addr learned my addr 172.21.15.89:0/894888409 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10dbfff700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc15bed0 0x7f10dc06fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.289+0000 7f10e0e26700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc0d1a50 0x7f10dc14b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10e1627700 1 -- 172.21.15.89:0/894888409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc15bed0 msgr2=0x7f10dc06fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10e1627700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc15bed0 0x7f10dc06fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10e1627700 1 -- 172.21.15.89:0/894888409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc0d1a50 msgr2=0x7f10dc14b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10e1627700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc0d1a50 0x7f10dc14b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10e1627700 1 -- 172.21.15.89:0/894888409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d0009550 con 0x7f10dc1603c0 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10e1627700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc1603c0 0x7f10dc070060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10d400cba0 tx=0x7f10d400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10d97fa700 1 -- 172.21.15.89:0/894888409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d400ab80 con 0x7f10dc1603c0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10d97fa700 1 -- 172.21.15.89:0/894888409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d400ace0 con 0x7f10dc1603c0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc1f1ac0 con 0x7f10dc1603c0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10d97fa700 1 -- 172.21.15.89:0/894888409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4012660 con 0x7f10dc1603c0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.290+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc1f2080 con 0x7f10dc1603c0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.291+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc067f60 con 0x7f10dc1603c0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.291+0000 7f10d97fa700 1 -- 172.21.15.89:0/894888409 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f10d40127c0 con 0x7f10dc1603c0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.291+0000 7f10d97fa700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f10c006aed0 0x7f10c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.292+0000 7f10d97fa700 1 -- 172.21.15.89:0/894888409 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10d4089e00 con 0x7f10dc1603c0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.292+0000 7f10e0e26700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f10c006aed0 0x7f10c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.292+0000 7f10e0e26700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f10c006aed0 0x7f10c006d390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009130 tx=0x7f10d0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:23.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.296+0000 7f10d97fa700 1 -- 172.21.15.89:0/894888409 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10d4054eb0 con 0x7f10dc1603c0 2023-11-24T06:19:23.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.505+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f10dc0b7bd0 con 0x7f10dc1603c0 2023-11-24T06:19:23.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.506+0000 7f10d97fa700 1 -- 172.21.15.89:0/894888409 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f10d4016020 con 0x7f10dc1603c0 2023-11-24T06:19:23.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f10c006aed0 msgr2=0x7f10c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f10c006aed0 0x7f10c006d390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009130 tx=0x7f10d0008030 comp rx=0 tx=0).stop 2023-11-24T06:19:23.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc1603c0 msgr2=0x7f10dc070060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc1603c0 0x7f10dc070060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10d400cba0 tx=0x7f10d400cf60 comp rx=0 tx=0).stop 2023-11-24T06:19:23.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 shutdown_connections 2023-11-24T06:19:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc1603c0 0x7f10dc070060 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f10c006aed0 0x7f10c006d390 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc0d1a50 0x7f10dc14b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 --2- 172.21.15.89:0/894888409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc15bed0 0x7f10dc06fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 >> 172.21.15.89:0/894888409 conn(0x7f10dc073130 msgr2=0x7f10dc0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 shutdown_connections 2023-11-24T06:19:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.508+0000 7f10e308a700 1 -- 172.21.15.89:0/894888409 wait complete. 2023-11-24T06:19:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2023-11-24T06:19:23.523 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T06:19:23.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:23.523 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-11-24T06:19:23.566 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:19:23.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:23.567 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T06:19:23.631 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:23.631 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:23.652 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:23.653 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:23.713 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:23.713 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:23.774 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:23.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.894+0000 7ff40bdb8700 1 Processor -- start 2023-11-24T06:19:23.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.894+0000 7ff40bdb8700 1 -- start start 2023-11-24T06:19:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff40bdb8700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4041446a0 0x7ff4040b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff40bdb8700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff4040adf00 0x7ff40406c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff40bdb8700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4040af950 con 0x7ff40406c540 2023-11-24T06:19:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff40bdb8700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40406c9f0 con 0x7ff4041446a0 2023-11-24T06:19:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff40bdb8700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff404144a80 con 0x7ff4040adf00 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409b54700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4041446a0 0x7ff4040b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409b54700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4041446a0 0x7ff4040b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36174/0 (socket says 172.21.15.89:36174) 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409b54700 1 -- 172.21.15.89:0/2612614170 learned_addr learned my addr 172.21.15.89:0/2612614170 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409353700 1 --2- 172.21.15.89:0/2612614170 >> v2:172.21.15.123:3300/0 conn(0x7ff4040adf00 0x7ff40406c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409b54700 1 -- 172.21.15.89:0/2612614170 >> v2:172.21.15.123:3300/0 conn(0x7ff4040adf00 msgr2=0x7ff40406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409b54700 1 --2- 172.21.15.89:0/2612614170 >> v2:172.21.15.123:3300/0 conn(0x7ff4040adf00 0x7ff40406c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409b54700 1 -- 172.21.15.89:0/2612614170 >> v1:172.21.15.123:6789/0 conn(0x7ff40406c540 legacy=0x7ff40406da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409b54700 1 -- 172.21.15.89:0/2612614170 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040afc30 con 0x7ff4041446a0 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409353700 1 --2- 172.21.15.89:0/2612614170 >> v2:172.21.15.123:3300/0 conn(0x7ff4040adf00 0x7ff40406c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.895+0000 7ff409b54700 1 --2- 172.21.15.89:0/2612614170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff4040b5ae0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff40409f030 tx=0x7ff400009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aaefba9015cf958 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff408b52700 1 -- 172.21.15.89:0/2612614170 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400004230 con 0x7ff4041446a0 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff408b52700 1 -- 172.21.15.89:0/2612614170 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff400004390 con 0x7ff4041446a0 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff408b52700 1 -- 172.21.15.89:0/2612614170 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4000036a0 con 0x7ff4041446a0 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2612614170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 msgr2=0x7ff4040b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2612614170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff4040b5ae0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff40409f030 tx=0x7ff400009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2612614170 shutdown_connections 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2612614170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff4040b5ae0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2612614170 >> v2:172.21.15.123:3300/0 conn(0x7ff4040adf00 0x7ff40406c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2612614170 >> 172.21.15.89:0/2612614170 conn(0x7ff404071b90 msgr2=0x7ff404071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:23.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2612614170 shutdown_connections 2023-11-24T06:19:23.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.896+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2612614170 wait complete. 2023-11-24T06:19:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.897+0000 7ff40bdb8700 1 Processor -- start 2023-11-24T06:19:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.897+0000 7ff40bdb8700 1 -- start start 2023-11-24T06:19:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40bdb8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff40406c540 0x7ff404157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40bdb8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4040adf00 0x7ff4041423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40bdb8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff40413b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40bdb8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff404143a60 con 0x7ff4041446a0 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40bdb8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff404142e50 con 0x7ff4040adf00 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40bdb8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff404143130 con 0x7ff40406c540 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40a355700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff40413b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40a355700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff40413b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36184/0 (socket says 172.21.15.89:36184) 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40a355700 1 -- 172.21.15.89:0/2826187515 learned_addr learned my addr 172.21.15.89:0/2826187515 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff409353700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4040adf00 0x7ff4041423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff409b54700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff40406c540 0x7ff404157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40a355700 1 -- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff40406c540 msgr2=0x7ff404157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40a355700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff40406c540 0x7ff404157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40a355700 1 -- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4040adf00 msgr2=0x7ff4041423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40a355700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4040adf00 0x7ff4041423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff40a355700 1 -- 172.21.15.89:0/2826187515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff400009550 con 0x7ff4041446a0 2023-11-24T06:19:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.898+0000 7ff409353700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4040adf00 0x7ff4041423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.899+0000 7ff40a355700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff40413b9f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00cc10 tx=0x7ff3fc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.899+0000 7ff3fa7fc700 1 -- 172.21.15.89:0/2826187515 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc00abb0 con 0x7ff4041446a0 2023-11-24T06:19:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.899+0000 7ff3fa7fc700 1 -- 172.21.15.89:0/2826187515 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc00ad10 con 0x7ff4041446a0 2023-11-24T06:19:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.899+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff404143410 con 0x7ff4041446a0 2023-11-24T06:19:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.899+0000 7ff3fa7fc700 1 -- 172.21.15.89:0/2826187515 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc012660 con 0x7ff4041446a0 2023-11-24T06:19:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.899+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4041584b0 con 0x7ff4041446a0 2023-11-24T06:19:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.900+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4040753f0 con 0x7ff4041446a0 2023-11-24T06:19:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.900+0000 7ff3fa7fc700 1 -- 172.21.15.89:0/2826187515 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff3fc0127c0 con 0x7ff4041446a0 2023-11-24T06:19:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.901+0000 7ff3fa7fc700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff3e806aed0 0x7ff3e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.901+0000 7ff3fa7fc700 1 -- 172.21.15.89:0/2826187515 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3fc08a2c0 con 0x7ff4041446a0 2023-11-24T06:19:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.901+0000 7ff409b54700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff3e806aed0 0x7ff3e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.901+0000 7ff409b54700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff3e806aed0 0x7ff3e806d390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff400009870 tx=0x7ff400008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:23.905+0000 7ff3fa7fc700 1 -- 172.21.15.89:0/2826187515 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3fc055450 con 0x7ff4041446a0 2023-11-24T06:19:24.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.096+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff404067f60 con 0x7ff4041446a0 2023-11-24T06:19:24.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.097+0000 7ff3fa7fc700 1 -- 172.21.15.89:0/2826187515 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7ff3fc058a70 con 0x7ff4041446a0 2023-11-24T06:19:24.100 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:24.101 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:24.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff3e806aed0 msgr2=0x7ff3e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff3e806aed0 0x7ff3e806d390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff400009870 tx=0x7ff400008030 comp rx=0 tx=0).stop 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 msgr2=0x7ff40413b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff40413b9f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00cc10 tx=0x7ff3fc00a060 comp rx=0 tx=0).stop 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 shutdown_connections 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4041446a0 0x7ff40413b9f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff40406c540 0x7ff404157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff3e806aed0 0x7ff3e806d390 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 --2- 172.21.15.89:0/2826187515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4040adf00 0x7ff4041423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.099+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 >> 172.21.15.89:0/2826187515 conn(0x7ff404071b90 msgr2=0x7ff4040b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:24.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.100+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 shutdown_connections 2023-11-24T06:19:24.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.100+0000 7ff40bdb8700 1 -- 172.21.15.89:0/2826187515 wait complete. 2023-11-24T06:19:24.115 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:24.115 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:24.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.237+0000 7f2e3751e700 1 Processor -- start 2023-11-24T06:19:24.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.238+0000 7f2e3751e700 1 -- start start 2023-11-24T06:19:24.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e3751e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e301587f0 0x7f2e3006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e3751e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2e3015b370 0x7f2e3015b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e3751e700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3006b530 con 0x7f2e300b3c60 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e3751e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300ae5e0 con 0x7f2e3015b370 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e3751e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3015be30 con 0x7f2e301587f0 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e352ba700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e301587f0 0x7f2e3006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e34ab9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2e3015b370 0x7f2e3015b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e352ba700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e301587f0 0x7f2e3006aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:45064/0 (socket says 172.21.15.89:45064) 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e35abb700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2e300b3c60 0x7f2e300af060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:43516/0 (socket says 172.21.15.89:43516) 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e352ba700 1 -- 172.21.15.89:0/1666227554 learned_addr learned my addr 172.21.15.89:0/1666227554 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e34ab9700 1 --2- 172.21.15.89:0/1666227554 >> v2:172.21.15.123:3300/0 conn(0x7f2e3015b370 0x7f2e3015b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:51300/0 (socket says 172.21.15.89:51300) 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e34ab9700 1 -- 172.21.15.89:0/1666227554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e301587f0 msgr2=0x7f2e3006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e34ab9700 1 --2- 172.21.15.89:0/1666227554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e301587f0 0x7f2e3006aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e34ab9700 1 -- 172.21.15.89:0/1666227554 >> v1:172.21.15.123:6789/0 conn(0x7f2e300b3c60 legacy=0x7f2e300af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e34ab9700 1 -- 172.21.15.89:0/1666227554 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3015c110 con 0x7f2e3015b370 2023-11-24T06:19:24.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.239+0000 7f2e352ba700 1 --2- 172.21.15.89:0/1666227554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e301587f0 0x7f2e3006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:24.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e34ab9700 1 --2- 172.21.15.89:0/1666227554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3015b370 0x7f2e3015b7f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009840 tx=0x7f2e2c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35bcfa52bcb98f6a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e27fff700 1 -- 172.21.15.89:0/1666227554 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c011830 con 0x7f2e3015b370 2023-11-24T06:19:24.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e27fff700 1 -- 172.21.15.89:0/1666227554 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2c011990 con 0x7f2e3015b370 2023-11-24T06:19:24.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e27fff700 1 -- 172.21.15.89:0/1666227554 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c011d20 con 0x7f2e3015b370 2023-11-24T06:19:24.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e3751e700 1 -- 172.21.15.89:0/1666227554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3015b370 msgr2=0x7f2e3015b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e3751e700 1 --2- 172.21.15.89:0/1666227554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3015b370 0x7f2e3015b7f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009840 tx=0x7f2e2c009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:24.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e3751e700 1 -- 172.21.15.89:0/1666227554 shutdown_connections 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e3751e700 1 --2- 172.21.15.89:0/1666227554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e301587f0 0x7f2e3006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e3751e700 1 --2- 172.21.15.89:0/1666227554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3015b370 0x7f2e3015b7f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e3751e700 1 -- 172.21.15.89:0/1666227554 >> 172.21.15.89:0/1666227554 conn(0x7f2e300d0bf0 msgr2=0x7f2e300d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e3751e700 1 -- 172.21.15.89:0/1666227554 shutdown_connections 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.240+0000 7f2e3751e700 1 -- 172.21.15.89:0/1666227554 wait complete. 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e3751e700 1 Processor -- start 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e3751e700 1 -- start start 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e3751e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300b3c60 0x7f2e3006c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e3751e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e301587f0 0x7f2e3009aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e3751e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015b370 0x7f2e301551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e3751e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30145e70 con 0x7f2e3015b370 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e3751e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301f0290 con 0x7f2e301587f0 2023-11-24T06:19:24.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e3751e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301ef680 con 0x7f2e300b3c60 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e35abb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015b370 0x7f2e301551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e35abb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015b370 0x7f2e301551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36196/0 (socket says 172.21.15.89:36196) 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.242+0000 7f2e35abb700 1 -- 172.21.15.89:0/2698914912 learned_addr learned my addr 172.21.15.89:0/2698914912 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e352ba700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300b3c60 0x7f2e3006c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e34ab9700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e301587f0 0x7f2e3009aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e35abb700 1 -- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300b3c60 msgr2=0x7f2e3006c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e35abb700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300b3c60 0x7f2e3006c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e35abb700 1 -- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e301587f0 msgr2=0x7f2e3009aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e35abb700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e301587f0 0x7f2e3009aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e35abb700 1 -- 172.21.15.89:0/2698914912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2000b7f0 con 0x7f2e3015b370 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e34ab9700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e301587f0 0x7f2e3009aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e35abb700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015b370 0x7f2e301551c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800cbd0 tx=0x7f2e2800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e25ffb700 1 -- 172.21.15.89:0/2698914912 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2800abb0 con 0x7f2e3015b370 2023-11-24T06:19:24.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e25ffb700 1 -- 172.21.15.89:0/2698914912 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2800ad10 con 0x7f2e3015b370 2023-11-24T06:19:24.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c0094a0 con 0x7f2e3015b370 2023-11-24T06:19:24.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e25ffb700 1 -- 172.21.15.89:0/2698914912 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28007270 con 0x7f2e3015b370 2023-11-24T06:19:24.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.243+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e301efc40 con 0x7f2e3015b370 2023-11-24T06:19:24.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.244+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e301f0430 con 0x7f2e3015b370 2023-11-24T06:19:24.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.244+0000 7f2e25ffb700 1 -- 172.21.15.89:0/2698914912 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2e280073d0 con 0x7f2e3015b370 2023-11-24T06:19:24.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.245+0000 7f2e25ffb700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2e1406aed0 0x7f2e1406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.245+0000 7f2e25ffb700 1 -- 172.21.15.89:0/2698914912 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e28089290 con 0x7f2e3015b370 2023-11-24T06:19:24.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.245+0000 7f2e352ba700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2e1406aed0 0x7f2e1406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.245+0000 7f2e352ba700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2e1406aed0 0x7f2e1406d390 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000bf90 tx=0x7f2e2000b4d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.249+0000 7f2e25ffb700 1 -- 172.21.15.89:0/2698914912 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e28054480 con 0x7f2e3015b370 2023-11-24T06:19:24.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.438+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e30068fb0 con 0x7f2e3015b370 2023-11-24T06:19:24.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.439+0000 7f2e25ffb700 1 -- 172.21.15.89:0/2698914912 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e28017020 con 0x7f2e3015b370 2023-11-24T06:19:24.442 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2e1406aed0 msgr2=0x7f2e1406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2e1406aed0 0x7f2e1406d390 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000bf90 tx=0x7f2e2000b4d0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015b370 msgr2=0x7f2e301551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015b370 0x7f2e301551c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800cbd0 tx=0x7f2e2800cf90 comp rx=0 tx=0).stop 2023-11-24T06:19:24.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 shutdown_connections 2023-11-24T06:19:24.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015b370 0x7f2e301551c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300b3c60 0x7f2e3006c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2e1406aed0 0x7f2e1406d390 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 --2- 172.21.15.89:0/2698914912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e301587f0 0x7f2e3009aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 >> 172.21.15.89:0/2698914912 conn(0x7f2e300d0bf0 msgr2=0x7f2e300aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:24.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 shutdown_connections 2023-11-24T06:19:24.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.441+0000 7f2e3751e700 1 -- 172.21.15.89:0/2698914912 wait complete. 2023-11-24T06:19:24.456 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:24.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.576+0000 7f0bd1adb700 1 Processor -- start 2023-11-24T06:19:24.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bd1adb700 1 -- start start 2023-11-24T06:19:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bd1adb700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bcc15b370 0x7f0bcc15b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bd1adb700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0bcc0b3c60 0x7f0bcc0b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bd1adb700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc0ae5e0 con 0x7f0bcc1587f0 2023-11-24T06:19:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bd1adb700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc0b4630 con 0x7f0bcc15b370 2023-11-24T06:19:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bd1adb700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc15be30 con 0x7f0bcc0b3c60 2023-11-24T06:19:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bc2ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bcc15b370 0x7f0bcc15b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bc2ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bcc15b370 0x7f0bcc15b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36200/0 (socket says 172.21.15.89:36200) 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.577+0000 7f0bc2ffd700 1 -- 172.21.15.89:0/2574155597 learned_addr learned my addr 172.21.15.89:0/2574155597 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bcbfff700 1 --2- 172.21.15.89:0/2574155597 >> v2:172.21.15.123:3300/0 conn(0x7f0bcc0b3c60 0x7f0bcc0b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bc2ffd700 1 -- 172.21.15.89:0/2574155597 >> v2:172.21.15.123:3300/0 conn(0x7f0bcc0b3c60 msgr2=0x7f0bcc0b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bc2ffd700 1 --2- 172.21.15.89:0/2574155597 >> v2:172.21.15.123:3300/0 conn(0x7f0bcc0b3c60 0x7f0bcc0b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bc2ffd700 1 -- 172.21.15.89:0/2574155597 >> v1:172.21.15.123:6789/0 conn(0x7f0bcc1587f0 legacy=0x7f0bcc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bc2ffd700 1 -- 172.21.15.89:0/2574155597 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc15c110 con 0x7f0bcc15b370 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bc2ffd700 1 --2- 172.21.15.89:0/2574155597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc15b800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40098f0 tx=0x7f0bb4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fe676ff87b4b2e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bcaffd700 1 -- 172.21.15.89:0/2574155597 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4011830 con 0x7f0bcc15b370 2023-11-24T06:19:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bcaffd700 1 -- 172.21.15.89:0/2574155597 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb4011990 con 0x7f0bcc15b370 2023-11-24T06:19:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.578+0000 7f0bcaffd700 1 -- 172.21.15.89:0/2574155597 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4011d20 con 0x7f0bcc15b370 2023-11-24T06:19:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.579+0000 7f0bd1adb700 1 -- 172.21.15.89:0/2574155597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 msgr2=0x7f0bcc15b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.579+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/2574155597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc15b800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40098f0 tx=0x7f0bb4009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.579+0000 7f0bd1adb700 1 -- 172.21.15.89:0/2574155597 shutdown_connections 2023-11-24T06:19:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.579+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/2574155597 >> v2:172.21.15.123:3300/0 conn(0x7f0bcc0b3c60 0x7f0bcc0b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.579+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/2574155597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc15b800 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.579+0000 7f0bd1adb700 1 -- 172.21.15.89:0/2574155597 >> 172.21.15.89:0/2574155597 conn(0x7f0bcc0d0bf0 msgr2=0x7f0bcc0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.579+0000 7f0bd1adb700 1 -- 172.21.15.89:0/2574155597 shutdown_connections 2023-11-24T06:19:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.579+0000 7f0bd1adb700 1 -- 172.21.15.89:0/2574155597 wait complete. 2023-11-24T06:19:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.580+0000 7f0bd1adb700 1 Processor -- start 2023-11-24T06:19:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.580+0000 7f0bd1adb700 1 -- start start 2023-11-24T06:19:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bd1adb700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc0b3c60 0x7f0bcc1415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bd1adb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc1587f0 0x7f0bcc0b1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bd1adb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bd1adb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc154c60 con 0x7f0bcc15b370 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bd1adb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc154020 con 0x7f0bcc1587f0 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bd1adb700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc1542c0 con 0x7f0bcc0b3c60 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36210/0 (socket says 172.21.15.89:36210) 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 -- 172.21.15.89:0/3736884541 learned_addr learned my addr 172.21.15.89:0/3736884541 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bc2ffd700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc1587f0 0x7f0bcc0b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcb7fe700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc0b3c60 0x7f0bcc1415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 -- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc0b3c60 msgr2=0x7f0bcc1415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc0b3c60 0x7f0bcc1415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 -- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc1587f0 msgr2=0x7f0bcc0b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc1587f0 0x7f0bcc0b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 -- 172.21.15.89:0/3736884541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb4009550 con 0x7f0bcc15b370 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bc2ffd700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc1587f0 0x7f0bcc0b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bcbfff700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc159d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400e950 tx=0x7f0bc400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bc8ff9700 1 -- 172.21.15.89:0/3736884541 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4009730 con 0x7f0bcc15b370 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bc8ff9700 1 -- 172.21.15.89:0/3736884541 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc400c850 con 0x7f0bcc15b370 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc1545a0 con 0x7f0bcc15b370 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.581+0000 7f0bc8ff9700 1 -- 172.21.15.89:0/3736884541 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc400cbe0 con 0x7f0bcc15b370 2023-11-24T06:19:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.582+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc141ae0 con 0x7f0bcc15b370 2023-11-24T06:19:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.582+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc154da0 con 0x7f0bcc15b370 2023-11-24T06:19:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.583+0000 7f0bc8ff9700 1 -- 172.21.15.89:0/3736884541 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0bc400cd40 con 0x7f0bcc15b370 2023-11-24T06:19:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.583+0000 7f0bc8ff9700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bac06aed0 0x7f0bac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.583+0000 7f0bc8ff9700 1 -- 172.21.15.89:0/3736884541 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bc40896c0 con 0x7f0bcc15b370 2023-11-24T06:19:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.583+0000 7f0bcb7fe700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bac06aed0 0x7f0bac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.584+0000 7f0bcb7fe700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bac06aed0 0x7f0bac06d390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc002140 tx=0x7f0bbc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.588+0000 7f0bc8ff9700 1 -- 172.21.15.89:0/3736884541 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bcc154da0 con 0x7f0bcc15b370 2023-11-24T06:19:24.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.778+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0bcc154da0 con 0x7f0bcc15b370 2023-11-24T06:19:24.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.779+0000 7f0bc8ff9700 1 -- 172.21.15.89:0/3736884541 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0bcc154da0 con 0x7f0bcc15b370 2023-11-24T06:19:24.782 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:24.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bac06aed0 msgr2=0x7f0bac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bac06aed0 0x7f0bac06d390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc002140 tx=0x7f0bbc009040 comp rx=0 tx=0).stop 2023-11-24T06:19:24.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 msgr2=0x7f0bcc159d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc159d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400e950 tx=0x7f0bc400ed10 comp rx=0 tx=0).stop 2023-11-24T06:19:24.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 shutdown_connections 2023-11-24T06:19:24.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc159d50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc0b3c60 0x7f0bcc1415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bac06aed0 0x7f0bac06d390 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 --2- 172.21.15.89:0/3736884541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc1587f0 0x7f0bcc0b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 >> 172.21.15.89:0/3736884541 conn(0x7f0bcc0d0bf0 msgr2=0x7f0bcc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:24.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 shutdown_connections 2023-11-24T06:19:24.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.781+0000 7f0bd1adb700 1 -- 172.21.15.89:0/3736884541 wait complete. 2023-11-24T06:19:24.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.915+0000 7f561bef0700 1 Processor -- start 2023-11-24T06:19:24.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.916+0000 7f561bef0700 1 -- start start 2023-11-24T06:19:24.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.916+0000 7f561bef0700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5614151660 0x7f56140cf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.916+0000 7f561bef0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f5614070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561bef0700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140b2b00 con 0x7f5614071080 2023-11-24T06:19:24.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561bef0700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f561406ee40 con 0x7f5614151660 2023-11-24T06:19:24.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561bef0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561406e3a0 con 0x7f561406dec0 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561a48d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5614071080 0x7f561415e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39722/0 (socket says 172.21.15.89:39722) 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561a48d700 1 -- 172.21.15.89:0/3794469043 learned_addr learned my addr 172.21.15.89:0/3794469043 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561948b700 1 --2- 172.21.15.89:0/3794469043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f5614070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f5619c8c700 1 --2- 172.21.15.89:0/3794469043 >> v2:172.21.15.123:3300/0 conn(0x7f5614151660 0x7f56140cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f5618c8a700 1 -- 172.21.15.89:0/3794469043 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010042149 0 0) 0x7f56140b2b00 con 0x7f5614071080 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f5618c8a700 1 -- 172.21.15.89:0/3794469043 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56000035b0 con 0x7f5614071080 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561948b700 1 -- 172.21.15.89:0/3794469043 >> v2:172.21.15.123:3300/0 conn(0x7f5614151660 msgr2=0x7f56140cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561948b700 1 --2- 172.21.15.89:0/3794469043 >> v2:172.21.15.123:3300/0 conn(0x7f5614151660 0x7f56140cf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561948b700 1 -- 172.21.15.89:0/3794469043 >> v1:172.21.15.89:6789/0 conn(0x7f5614071080 legacy=0x7f561415e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561948b700 1 -- 172.21.15.89:0/3794469043 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140b2da0 con 0x7f561406dec0 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f5619c8c700 1 --2- 172.21.15.89:0/3794469043 >> v2:172.21.15.123:3300/0 conn(0x7f5614151660 0x7f56140cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.917+0000 7f561948b700 1 --2- 172.21.15.89:0/3794469043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f5614070b40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5610009840 tx=0x7f5610009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d2c73f62e331585 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f5618c8a700 1 -- 172.21.15.89:0/3794469043 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610011830 con 0x7f561406dec0 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f5618c8a700 1 -- 172.21.15.89:0/3794469043 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5610011990 con 0x7f561406dec0 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f5618c8a700 1 -- 172.21.15.89:0/3794469043 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610011d00 con 0x7f561406dec0 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f561bef0700 1 -- 172.21.15.89:0/3794469043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 msgr2=0x7f5614070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f561bef0700 1 --2- 172.21.15.89:0/3794469043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f5614070b40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5610009840 tx=0x7f5610009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f561bef0700 1 -- 172.21.15.89:0/3794469043 shutdown_connections 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f561bef0700 1 --2- 172.21.15.89:0/3794469043 >> v2:172.21.15.123:3300/0 conn(0x7f5614151660 0x7f56140cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f561bef0700 1 --2- 172.21.15.89:0/3794469043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f5614070b40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f561bef0700 1 -- 172.21.15.89:0/3794469043 >> 172.21.15.89:0/3794469043 conn(0x7f561406b410 msgr2=0x7f561406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:24.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f561bef0700 1 -- 172.21.15.89:0/3794469043 shutdown_connections 2023-11-24T06:19:24.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.918+0000 7f561bef0700 1 -- 172.21.15.89:0/3794469043 wait complete. 2023-11-24T06:19:24.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.919+0000 7f561bef0700 1 Processor -- start 2023-11-24T06:19:24.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.919+0000 7f561bef0700 1 -- start start 2023-11-24T06:19:24.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561bef0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f561413bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561bef0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5614071080 0x7f56140cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561bef0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5614151660 0x7f56140cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561bef0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5614159560 con 0x7f5614071080 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561bef0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5614159840 con 0x7f5614151660 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561bef0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5614159b20 con 0x7f561406dec0 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561948b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5614071080 0x7f56140cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561948b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5614071080 0x7f56140cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36222/0 (socket says 172.21.15.89:36222) 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561948b700 1 -- 172.21.15.89:0/3710487803 learned_addr learned my addr 172.21.15.89:0/3710487803 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f5619c8c700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f561413bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561a48d700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5614151660 0x7f56140cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561948b700 1 -- 172.21.15.89:0/3710487803 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 msgr2=0x7f561413bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561948b700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f561413bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561948b700 1 -- 172.21.15.89:0/3710487803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5614151660 msgr2=0x7f56140cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561948b700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5614151660 0x7f56140cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.920+0000 7f561948b700 1 -- 172.21.15.89:0/3710487803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56080094a0 con 0x7f5614071080 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.921+0000 7f561948b700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5614071080 0x7f56140cd160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5610009fd0 tx=0x7f5610004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.921+0000 7f56067fc700 1 -- 172.21.15.89:0/3710487803 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56100036a0 con 0x7f5614071080 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.921+0000 7f56067fc700 1 -- 172.21.15.89:0/3710487803 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561001f070 con 0x7f5614071080 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.921+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56100094a0 con 0x7f5614071080 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.921+0000 7f56067fc700 1 -- 172.21.15.89:0/3710487803 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610003a20 con 0x7f5614071080 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.921+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561415d740 con 0x7f5614071080 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.922+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5614068480 con 0x7f5614071080 2023-11-24T06:19:24.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.922+0000 7f56067fc700 1 -- 172.21.15.89:0/3710487803 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5610003b80 con 0x7f5614071080 2023-11-24T06:19:24.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.922+0000 7f56067fc700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f55f806aed0 0x7f55f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:24.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.923+0000 7f56067fc700 1 -- 172.21.15.89:0/3710487803 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5610021070 con 0x7f5614071080 2023-11-24T06:19:24.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.923+0000 7f5619c8c700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f55f806aed0 0x7f55f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:24.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.923+0000 7f5619c8c700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f55f806aed0 0x7f55f806d390 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5608009c60 tx=0x7f5608008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:24.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:24.927+0000 7f56067fc700 1 -- 172.21.15.89:0/3710487803 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5610090020 con 0x7f5614071080 2023-11-24T06:19:25.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.115+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f561415a040 con 0x7f5614071080 2023-11-24T06:19:25.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.116+0000 7f56067fc700 1 -- 172.21.15.89:0/3710487803 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f561415a040 con 0x7f5614071080 2023-11-24T06:19:25.119 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:25.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f55f806aed0 msgr2=0x7f55f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f55f806aed0 0x7f55f806d390 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5608009c60 tx=0x7f5608008040 comp rx=0 tx=0).stop 2023-11-24T06:19:25.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5614071080 msgr2=0x7f56140cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5614071080 0x7f56140cd160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5610009fd0 tx=0x7f5610004750 comp rx=0 tx=0).stop 2023-11-24T06:19:25.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 shutdown_connections 2023-11-24T06:19:25.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f55f806aed0 0x7f55f806d390 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5614151660 0x7f56140cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561406dec0 0x7f561413bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 --2- 172.21.15.89:0/3710487803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5614071080 0x7f56140cd160 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 >> 172.21.15.89:0/3710487803 conn(0x7f561406b410 msgr2=0x7f561415cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:25.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 shutdown_connections 2023-11-24T06:19:25.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.118+0000 7f561bef0700 1 -- 172.21.15.89:0/3710487803 wait complete. 2023-11-24T06:19:25.133 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:19:25.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.252+0000 7faa32b6a700 1 Processor -- start 2023-11-24T06:19:25.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.253+0000 7faa32b6a700 1 -- start start 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.253+0000 7faa32b6a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faa2c15e080 0x7faa2c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.253+0000 7faa32b6a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faa2c06c710 0x7faa2c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.253+0000 7faa32b6a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2c0ae430 con 0x7faa2c0af4d0 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.253+0000 7faa32b6a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c0af0b0 con 0x7faa2c15e080 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.253+0000 7faa32b6a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c0b05a0 con 0x7faa2c06c710 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.253+0000 7faa30906700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faa2c15e080 0x7faa2c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.253+0000 7faa2bfff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faa2c0af4d0 0x7faa2c06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39736/0 (socket says 172.21.15.89:39736) 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa2bfff700 1 -- 172.21.15.89:0/3101270907 learned_addr learned my addr 172.21.15.89:0/3101270907 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa30906700 1 -- 172.21.15.89:0/3101270907 >> v2:172.21.15.123:3300/0 conn(0x7faa2c06c710 msgr2=0x7faa2c0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:25.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa31107700 1 --2- 172.21.15.89:0/3101270907 >> v2:172.21.15.123:3300/0 conn(0x7faa2c06c710 0x7faa2c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa2b7fe700 1 -- 172.21.15.89:0/3101270907 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881150754 0 0) 0x7faa2c0ae430 con 0x7faa2c0af4d0 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa30906700 1 --2- 172.21.15.89:0/3101270907 >> v2:172.21.15.123:3300/0 conn(0x7faa2c06c710 0x7faa2c0afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa30906700 1 -- 172.21.15.89:0/3101270907 >> v1:172.21.15.89:6789/0 conn(0x7faa2c0af4d0 legacy=0x7faa2c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa30906700 1 -- 172.21.15.89:0/3101270907 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c0ae710 con 0x7faa2c15e080 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa30906700 1 --2- 172.21.15.89:0/3101270907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c15e080 0x7faa2c0b5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faa2c06e060 tx=0x7faa20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408ae013b8d008f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa2b7fe700 1 -- 172.21.15.89:0/3101270907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20011830 con 0x7faa2c15e080 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa2b7fe700 1 -- 172.21.15.89:0/3101270907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa20011990 con 0x7faa2c15e080 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.254+0000 7faa2b7fe700 1 -- 172.21.15.89:0/3101270907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20011d20 con 0x7faa2c15e080 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.255+0000 7faa32b6a700 1 -- 172.21.15.89:0/3101270907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c15e080 msgr2=0x7faa2c0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.255+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3101270907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c15e080 0x7faa2c0b5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faa2c06e060 tx=0x7faa20009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:25.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.255+0000 7faa32b6a700 1 -- 172.21.15.89:0/3101270907 shutdown_connections 2023-11-24T06:19:25.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.255+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3101270907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c15e080 0x7faa2c0b5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.255+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3101270907 >> v2:172.21.15.123:3300/0 conn(0x7faa2c06c710 0x7faa2c0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.255+0000 7faa32b6a700 1 -- 172.21.15.89:0/3101270907 >> 172.21.15.89:0/3101270907 conn(0x7faa2c071b90 msgr2=0x7faa2c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:25.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.255+0000 7faa32b6a700 1 -- 172.21.15.89:0/3101270907 shutdown_connections 2023-11-24T06:19:25.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.255+0000 7faa32b6a700 1 -- 172.21.15.89:0/3101270907 wait complete. 2023-11-24T06:19:25.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.256+0000 7faa32b6a700 1 Processor -- start 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.256+0000 7faa32b6a700 1 -- start start 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa32b6a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa2c06c710 0x7faa2c1f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa32b6a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c0af4d0 0x7faa2c14a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa32b6a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa2c15e080 0x7faa2c155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa32b6a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c155480 con 0x7faa2c0af4d0 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa32b6a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c15a180 con 0x7faa2c06c710 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa32b6a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c1f48d0 con 0x7faa2c15e080 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c0af4d0 0x7faa2c14a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c0af4d0 0x7faa2c14a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36236/0 (socket says 172.21.15.89:36236) 2023-11-24T06:19:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 -- 172.21.15.89:0/3454400699 learned_addr learned my addr 172.21.15.89:0/3454400699 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa31107700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa2c15e080 0x7faa2c155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa30906700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa2c06c710 0x7faa2c1f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 -- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa2c15e080 msgr2=0x7faa2c155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa2c15e080 0x7faa2c155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 -- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa2c06c710 msgr2=0x7faa2c1f4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa2c06c710 0x7faa2c1f4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 -- 172.21.15.89:0/3454400699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa200094a0 con 0x7faa2c0af4d0 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa30906700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa2c06c710 0x7faa2c1f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa2bfff700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c0af4d0 0x7faa2c14a520 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faa1800c570 tx=0x7faa1800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.257+0000 7faa297fa700 1 -- 172.21.15.89:0/3454400699 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18004c80 con 0x7faa2c0af4d0 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.258+0000 7faa297fa700 1 -- 172.21.15.89:0/3454400699 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18004e00 con 0x7faa2c0af4d0 2023-11-24T06:19:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.258+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2c155910 con 0x7faa2c0af4d0 2023-11-24T06:19:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.258+0000 7faa297fa700 1 -- 172.21.15.89:0/3454400699 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18007310 con 0x7faa2c0af4d0 2023-11-24T06:19:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.258+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c154b50 con 0x7faa2c0af4d0 2023-11-24T06:19:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.258+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2c0753f0 con 0x7faa2c0af4d0 2023-11-24T06:19:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.259+0000 7faa297fa700 1 -- 172.21.15.89:0/3454400699 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faa18007470 con 0x7faa2c0af4d0 2023-11-24T06:19:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.259+0000 7faa297fa700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faa1006aed0 0x7faa1006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.259+0000 7faa30906700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faa1006aed0 0x7faa1006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.259+0000 7faa297fa700 1 -- 172.21.15.89:0/3454400699 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa18089370 con 0x7faa2c0af4d0 2023-11-24T06:19:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.260+0000 7faa30906700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faa1006aed0 0x7faa1006d390 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0b53b0 tx=0x7faa20008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:25.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.264+0000 7faa297fa700 1 -- 172.21.15.89:0/3454400699 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa18054420 con 0x7faa2c0af4d0 2023-11-24T06:19:25.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.471+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faa2c155630 con 0x7faa2c0af4d0 2023-11-24T06:19:25.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.471+0000 7faa297fa700 1 -- 172.21.15.89:0/3454400699 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7faa2c155630 con 0x7faa2c0af4d0 2023-11-24T06:19:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faa1006aed0 msgr2=0x7faa1006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faa1006aed0 0x7faa1006d390 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0b53b0 tx=0x7faa20008030 comp rx=0 tx=0).stop 2023-11-24T06:19:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c0af4d0 msgr2=0x7faa2c14a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c0af4d0 0x7faa2c14a520 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faa1800c570 tx=0x7faa1800c930 comp rx=0 tx=0).stop 2023-11-24T06:19:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 shutdown_connections 2023-11-24T06:19:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa2c15e080 0x7faa2c155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faa1006aed0 0x7faa1006d390 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa2c06c710 0x7faa2c1f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 --2- 172.21.15.89:0/3454400699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa2c0af4d0 0x7faa2c14a520 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 >> 172.21.15.89:0/3454400699 conn(0x7faa2c071b90 msgr2=0x7faa2c15d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 shutdown_connections 2023-11-24T06:19:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.474+0000 7faa32b6a700 1 -- 172.21.15.89:0/3454400699 wait complete. 2023-11-24T06:19:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2023-11-24T06:19:25.489 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T06:19:25.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:25.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-11-24T06:19:25.536 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:19:25.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:25.536 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T06:19:25.568 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:25.568 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:25.586 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:25.586 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:25.644 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:25.644 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:25.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:25.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.857+0000 7f57af80d700 1 Processor -- start 2023-11-24T06:19:25.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.857+0000 7f57af80d700 1 -- start start 2023-11-24T06:19:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.858+0000 7f57af80d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f57a806c710 0x7f57a8165b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.858+0000 7f57af80d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a80af950 con 0x7f57a81446a0 2023-11-24T06:19:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.858+0000 7f57af80d700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a80afc30 con 0x7f57a80adf00 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.858+0000 7f57af80d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a8144a80 con 0x7f57a806c710 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.858+0000 7f57ad5a9700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f57a81446a0 0x7f57a80b5ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39746/0 (socket says 172.21.15.89:39746) 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.858+0000 7f57ad5a9700 1 -- 172.21.15.89:0/3449885116 learned_addr learned my addr 172.21.15.89:0/3449885116 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57addaa700 1 --2- 172.21.15.89:0/3449885116 >> v2:172.21.15.123:3300/0 conn(0x7f57a806c710 0x7f57a8165b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293889347 0 0) 0x7f57a80af950 con 0x7f57a81446a0 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57940035b0 con 0x7f57a81446a0 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 833625916 0 0) 0x7f57940035b0 con 0x7f57a81446a0 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 >> v2:172.21.15.123:3300/0 conn(0x7f57a806c710 msgr2=0x7f57a8165b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 --2- 172.21.15.89:0/3449885116 >> v2:172.21.15.123:3300/0 conn(0x7f57a806c710 0x7f57a8165b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 >> v1:172.21.15.123:6789/0 conn(0x7f57a80adf00 legacy=0x7f57a806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a80aff10 con 0x7f57a81446a0 2023-11-24T06:19:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f5798001670 con 0x7f57a81446a0 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 >> v1:172.21.15.89:6789/0 conn(0x7f57a81446a0 legacy=0x7f57a80b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f5794003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5794004850 0x7f5794004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5794005210 0x7f579400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a8144a80 con 0x7f5794003a00 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579400c1b0 con 0x7f5794005210 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579400c370 con 0x7f5794004850 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57ad5a9700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f5794003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57addaa700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5794005210 0x7f579400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57acda8700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5794004850 0x7f5794004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57ad5a9700 1 -- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5794004850 msgr2=0x7f5794004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57ad5a9700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5794004850 0x7f5794004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57ad5a9700 1 -- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5794005210 msgr2=0x7f579400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57ad5a9700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5794005210 0x7f579400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57addaa700 1 -- 172.21.15.89:0/3449885116 reap_dead start 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57ad5a9700 1 -- 172.21.15.89:0/3449885116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57940035b0 con 0x7f5794003a00 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.859+0000 7f57acda8700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5794004850 0x7f5794004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f57ad5a9700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f5794003e20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5798007d30 tx=0x7f5794005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1a03be8370f7b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579800cd30 con 0x7f5794003a00 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579800ce90 con 0x7f5794003a00 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f579ffff700 1 -- 172.21.15.89:0/3449885116 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798005ba0 con 0x7f5794003a00 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f57af80d700 1 -- 172.21.15.89:0/3449885116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 msgr2=0x7f5794003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f57af80d700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f5794003e20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5798007d30 tx=0x7f5794005880 comp rx=0 tx=0).stop 2023-11-24T06:19:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f57af80d700 1 -- 172.21.15.89:0/3449885116 shutdown_connections 2023-11-24T06:19:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f57af80d700 1 --2- 172.21.15.89:0/3449885116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f5794003e20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f57af80d700 1 -- 172.21.15.89:0/3449885116 >> 172.21.15.89:0/3449885116 conn(0x7f57a8071b90 msgr2=0x7f57a8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f57af80d700 1 -- 172.21.15.89:0/3449885116 shutdown_connections 2023-11-24T06:19:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.860+0000 7f57af80d700 1 -- 172.21.15.89:0/3449885116 wait complete. 2023-11-24T06:19:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.861+0000 7f57af80d700 1 Processor -- start 2023-11-24T06:19:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.861+0000 7f57af80d700 1 -- start start 2023-11-24T06:19:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57af80d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f57a809e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57af80d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57a815a670 0x7f57a80cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57af80d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57a80cf130 0x7f57a809c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57af80d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a80cf5c0 con 0x7f5794003a00 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57af80d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a809d660 con 0x7f57a80cf130 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57af80d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a809d940 con 0x7f57a815a670 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57ad5a9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f57a809e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57ad5a9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f57a809e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36254/0 (socket says 172.21.15.89:36254) 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57ad5a9700 1 -- 172.21.15.89:0/594745054 learned_addr learned my addr 172.21.15.89:0/594745054 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57acda8700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57a815a670 0x7f57a80cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.862+0000 7f57addaa700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57a80cf130 0x7f57a809c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f57ad5a9700 1 -- 172.21.15.89:0/594745054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57a815a670 msgr2=0x7f57a80cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f57ad5a9700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57a815a670 0x7f57a80cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f57ad5a9700 1 -- 172.21.15.89:0/594745054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57a80cf130 msgr2=0x7f57a809c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f57ad5a9700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57a80cf130 0x7f57a809c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f57ad5a9700 1 -- 172.21.15.89:0/594745054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a809dc20 con 0x7f5794003a00 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f57ad5a9700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f57a809e700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f579800cde0 tx=0x7f5798001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f579dffb700 1 -- 172.21.15.89:0/594745054 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798001640 con 0x7f5794003a00 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f579dffb700 1 -- 172.21.15.89:0/594745054 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5798000ed0 con 0x7f5794003a00 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a809df00 con 0x7f5794003a00 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f579dffb700 1 -- 172.21.15.89:0/594745054 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798007330 con 0x7f5794003a00 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.863+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a815aaf0 con 0x7f5794003a00 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.864+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a80753f0 con 0x7f5794003a00 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.865+0000 7f579dffb700 1 -- 172.21.15.89:0/594745054 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5798007490 con 0x7f5794003a00 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.865+0000 7f579dffb700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f578c06aed0 0x7f578c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.865+0000 7f579dffb700 1 -- 172.21.15.89:0/594745054 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f579808c820 con 0x7f5794003a00 2023-11-24T06:19:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.865+0000 7f57acda8700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f578c06aed0 0x7f578c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.865+0000 7f57acda8700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f578c06aed0 0x7f578c06d390 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f57a4007fd0 tx=0x7f57a4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:25.869+0000 7f579dffb700 1 -- 172.21.15.89:0/594745054 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5798057960 con 0x7f5794003a00 2023-11-24T06:19:26.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.060+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57a8067f60 con 0x7f5794003a00 2023-11-24T06:19:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.061+0000 7f579dffb700 1 -- 172.21.15.89:0/594745054 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f5798015090 con 0x7f5794003a00 2023-11-24T06:19:26.064 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:26.065 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f578c06aed0 msgr2=0x7f578c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f578c06aed0 0x7f578c06d390 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f57a4007fd0 tx=0x7f57a4009040 comp rx=0 tx=0).stop 2023-11-24T06:19:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 msgr2=0x7f57a809e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f57a809e700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f579800cde0 tx=0x7f5798001a00 comp rx=0 tx=0).stop 2023-11-24T06:19:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 shutdown_connections 2023-11-24T06:19:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f578c06aed0 0x7f578c06d390 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57a80cf130 0x7f57a809c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5794003a00 0x7f57a809e700 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.063+0000 7f57af80d700 1 --2- 172.21.15.89:0/594745054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57a815a670 0x7f57a80cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.064+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 >> 172.21.15.89:0/594745054 conn(0x7f57a8071b90 msgr2=0x7f57a80b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.064+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 shutdown_connections 2023-11-24T06:19:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.064+0000 7f57af80d700 1 -- 172.21.15.89:0/594745054 wait complete. 2023-11-24T06:19:26.079 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:26.079 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:26.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f9663700 1 Processor -- start 2023-11-24T06:19:26.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f9663700 1 -- start start 2023-11-24T06:19:26.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f9663700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69f40d39d0 0x7f69f40b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f9663700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40d2930 con 0x7f69f415e140 2023-11-24T06:19:26.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f9663700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40d2c10 con 0x7f69f40d45d0 2023-11-24T06:19:26.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f9663700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f415e520 con 0x7f69f40d39d0 2023-11-24T06:19:26.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f2ffd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f69f415e140 0x7f69f40d3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39766/0 (socket says 172.21.15.89:39766) 2023-11-24T06:19:26.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f2ffd700 1 -- 172.21.15.89:0/1479762013 learned_addr learned my addr 172.21.15.89:0/1479762013 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:26.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f37fe700 1 --2- 172.21.15.89:0/1479762013 >> v2:172.21.15.89:3300/0 conn(0x7f69f40d39d0 0x7f69f40b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f1ffb700 1 -- 172.21.15.89:0/1479762013 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80137033 0 0) 0x7f69f40d2930 con 0x7f69f415e140 2023-11-24T06:19:26.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f1ffb700 1 -- 172.21.15.89:0/1479762013 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc0035b0 con 0x7f69f415e140 2023-11-24T06:19:26.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f37fe700 1 -- 172.21.15.89:0/1479762013 >> v1:172.21.15.123:6789/0 conn(0x7f69f40d45d0 legacy=0x7f69f40b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:26.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f37fe700 1 -- 172.21.15.89:0/1479762013 >> v1:172.21.15.89:6789/0 conn(0x7f69f415e140 legacy=0x7f69f40d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f37fe700 1 -- 172.21.15.89:0/1479762013 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f40d2ef0 con 0x7f69f40d39d0 2023-11-24T06:19:26.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f37fe700 1 --2- 172.21.15.89:0/1479762013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d39d0 0x7f69f40b77e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009d90 tx=0x7f69ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc6b9ffd8a28164 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f1ffb700 1 -- 172.21.15.89:0/1479762013 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec00e070 con 0x7f69f40d39d0 2023-11-24T06:19:26.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f1ffb700 1 -- 172.21.15.89:0/1479762013 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec00bae0 con 0x7f69f40d39d0 2023-11-24T06:19:26.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.202+0000 7f69f1ffb700 1 -- 172.21.15.89:0/1479762013 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec00be90 con 0x7f69f40d39d0 2023-11-24T06:19:26.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.203+0000 7f69f9663700 1 -- 172.21.15.89:0/1479762013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d39d0 msgr2=0x7f69f40b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.203+0000 7f69f9663700 1 --2- 172.21.15.89:0/1479762013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d39d0 0x7f69f40b77e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009d90 tx=0x7f69ec00b330 comp rx=0 tx=0).stop 2023-11-24T06:19:26.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.203+0000 7f69f9663700 1 -- 172.21.15.89:0/1479762013 shutdown_connections 2023-11-24T06:19:26.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.203+0000 7f69f9663700 1 --2- 172.21.15.89:0/1479762013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d39d0 0x7f69f40b77e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.203+0000 7f69f9663700 1 -- 172.21.15.89:0/1479762013 >> 172.21.15.89:0/1479762013 conn(0x7f69f406b410 msgr2=0x7f69f406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:26.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.203+0000 7f69f9663700 1 -- 172.21.15.89:0/1479762013 shutdown_connections 2023-11-24T06:19:26.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.203+0000 7f69f9663700 1 -- 172.21.15.89:0/1479762013 wait complete. 2023-11-24T06:19:26.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.204+0000 7f69f9663700 1 Processor -- start 2023-11-24T06:19:26.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.204+0000 7f69f9663700 1 -- start start 2023-11-24T06:19:26.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f9663700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69f40d39d0 0x7f69f40cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f9663700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d45d0 0x7f69f4157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f9663700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69f415e140 0x7f69f415f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f9663700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4157ad0 con 0x7f69f40d45d0 2023-11-24T06:19:26.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f9663700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40b4aa0 con 0x7f69f415e140 2023-11-24T06:19:26.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f9663700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40b4d80 con 0x7f69f40d39d0 2023-11-24T06:19:26.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f27fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d45d0 0x7f69f4157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f27fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d45d0 0x7f69f4157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36270/0 (socket says 172.21.15.89:36270) 2023-11-24T06:19:26.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f27fc700 1 -- 172.21.15.89:0/1727410191 learned_addr learned my addr 172.21.15.89:0/1727410191 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:26.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f2ffd700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69f40d39d0 0x7f69f40cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f37fe700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69f415e140 0x7f69f415f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f27fc700 1 -- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69f40d39d0 msgr2=0x7f69f40cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f27fc700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69f40d39d0 0x7f69f40cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.205+0000 7f69f27fc700 1 -- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69f415e140 msgr2=0x7f69f415f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69f27fc700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69f415e140 0x7f69f415f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69f27fc700 1 -- 172.21.15.89:0/1727410191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec009a70 con 0x7f69f40d45d0 2023-11-24T06:19:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69f37fe700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69f415e140 0x7f69f415f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:26.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69f27fc700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d45d0 0x7f69f4157590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c4d0 tx=0x7f69e800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69db7fe700 1 -- 172.21.15.89:0/1727410191 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8011790 con 0x7f69f40d45d0 2023-11-24T06:19:26.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69db7fe700 1 -- 172.21.15.89:0/1727410191 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e80118f0 con 0x7f69f40d45d0 2023-11-24T06:19:26.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f40b5060 con 0x7f69f40d45d0 2023-11-24T06:19:26.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69db7fe700 1 -- 172.21.15.89:0/1727410191 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8011ca0 con 0x7f69f40d45d0 2023-11-24T06:19:26.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.206+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f40b5620 con 0x7f69f40d45d0 2023-11-24T06:19:26.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.207+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f40b57b0 con 0x7f69f40d45d0 2023-11-24T06:19:26.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.207+0000 7f69db7fe700 1 -- 172.21.15.89:0/1727410191 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f69e801c460 con 0x7f69f40d45d0 2023-11-24T06:19:26.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.208+0000 7f69db7fe700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f69e006aed0 0x7f69e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.208+0000 7f69f2ffd700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f69e006aed0 0x7f69e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.208+0000 7f69db7fe700 1 -- 172.21.15.89:0/1727410191 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69e8089350 con 0x7f69f40d45d0 2023-11-24T06:19:26.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.208+0000 7f69f2ffd700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f69e006aed0 0x7f69e006d390 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f69e4007fd0 tx=0x7f69e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.213+0000 7f69db7fe700 1 -- 172.21.15.89:0/1727410191 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69e8054400 con 0x7f69f40d45d0 2023-11-24T06:19:26.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.401+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f69f4068fb0 con 0x7f69f40d45d0 2023-11-24T06:19:26.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.402+0000 7f69db7fe700 1 -- 172.21.15.89:0/1727410191 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f69e8057a20 con 0x7f69f40d45d0 2023-11-24T06:19:26.405 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f69e006aed0 msgr2=0x7f69e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f69e006aed0 0x7f69e006d390 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f69e4007fd0 tx=0x7f69e4009040 comp rx=0 tx=0).stop 2023-11-24T06:19:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d45d0 msgr2=0x7f69f4157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d45d0 0x7f69f4157590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c4d0 tx=0x7f69e800c7e0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 shutdown_connections 2023-11-24T06:19:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f69e006aed0 0x7f69e006d390 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69f415e140 0x7f69f415f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69f40d39d0 0x7f69f40cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 --2- 172.21.15.89:0/1727410191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f40d45d0 0x7f69f4157590 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 >> 172.21.15.89:0/1727410191 conn(0x7f69f406b410 msgr2=0x7f69f40d3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 shutdown_connections 2023-11-24T06:19:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.404+0000 7f69f9663700 1 -- 172.21.15.89:0/1727410191 wait complete. 2023-11-24T06:19:26.419 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79c0f62700 1 Processor -- start 2023-11-24T06:19:26.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79c0f62700 1 -- start start 2023-11-24T06:19:26.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79c0f62700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f79bc1587f0 0x7f79bc06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79c0f62700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79bc15b370 0x7f79bc15b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79c0f62700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 0x7f79bc0b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79c0f62700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79bc0b4630 con 0x7f79bc15b370 2023-11-24T06:19:26.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79c0f62700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79bc06b530 con 0x7f79bc1587f0 2023-11-24T06:19:26.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79c0f62700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc0ae5e0 con 0x7f79bc0b3c60 2023-11-24T06:19:26.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79bc15b370 0x7f79bc15b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79bc15b370 0x7f79bc15b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36278/0 (socket says 172.21.15.89:36278) 2023-11-24T06:19:26.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 -- 172.21.15.89:0/1188936144 learned_addr learned my addr 172.21.15.89:0/1188936144 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:26.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79bad9d700 1 --2- 172.21.15.89:0/1188936144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 0x7f79bc0b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 -- 172.21.15.89:0/1188936144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 msgr2=0x7f79bc0b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:26.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 --2- 172.21.15.89:0/1188936144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 0x7f79bc0b40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79ba59c700 1 --2- 172.21.15.89:0/1188936144 >> v2:172.21.15.123:3300/0 conn(0x7f79bc1587f0 0x7f79bc06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 -- 172.21.15.89:0/1188936144 >> v2:172.21.15.123:3300/0 conn(0x7f79bc1587f0 msgr2=0x7f79bc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:26.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 --2- 172.21.15.89:0/1188936144 >> v2:172.21.15.123:3300/0 conn(0x7f79bc1587f0 0x7f79bc06aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 -- 172.21.15.89:0/1188936144 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc15be30 con 0x7f79bc15b370 2023-11-24T06:19:26.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b9d9b700 1 --2- 172.21.15.89:0/1188936144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc15b800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f79b000bb90 tx=0x7f79b000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492596b4e0fd80f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.540+0000 7f79b959a700 1 -- 172.21.15.89:0/1188936144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b000dd10 con 0x7f79bc15b370 2023-11-24T06:19:26.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79b959a700 1 -- 172.21.15.89:0/1188936144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b000de70 con 0x7f79bc15b370 2023-11-24T06:19:26.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79b959a700 1 -- 172.21.15.89:0/1188936144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b0010670 con 0x7f79bc15b370 2023-11-24T06:19:26.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 -- 172.21.15.89:0/1188936144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 msgr2=0x7f79bc15b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 --2- 172.21.15.89:0/1188936144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc15b800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f79b000bb90 tx=0x7f79b000bea0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 -- 172.21.15.89:0/1188936144 shutdown_connections 2023-11-24T06:19:26.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 --2- 172.21.15.89:0/1188936144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 0x7f79bc0b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 --2- 172.21.15.89:0/1188936144 >> v2:172.21.15.123:3300/0 conn(0x7f79bc1587f0 0x7f79bc06aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 --2- 172.21.15.89:0/1188936144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc15b800 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 -- 172.21.15.89:0/1188936144 >> 172.21.15.89:0/1188936144 conn(0x7f79bc0d0bf0 msgr2=0x7f79bc0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:26.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 -- 172.21.15.89:0/1188936144 shutdown_connections 2023-11-24T06:19:26.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.541+0000 7f79c0f62700 1 -- 172.21.15.89:0/1188936144 wait complete. 2023-11-24T06:19:26.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.542+0000 7f79c0f62700 1 Processor -- start 2023-11-24T06:19:26.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79c0f62700 1 -- start start 2023-11-24T06:19:26.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79c0f62700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 0x7f79bc153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79c0f62700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79bc1587f0 0x7f79bc1f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79c0f62700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc0aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79c0f62700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc09baa0 con 0x7f79bc15b370 2023-11-24T06:19:26.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79c0f62700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc1f8460 con 0x7f79bc1587f0 2023-11-24T06:19:26.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79c0f62700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc09b7c0 con 0x7f79bc0b3c60 2023-11-24T06:19:26.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79bad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc0aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79bad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc0aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36286/0 (socket says 172.21.15.89:36286) 2023-11-24T06:19:26.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79bad9d700 1 -- 172.21.15.89:0/4057937618 learned_addr learned my addr 172.21.15.89:0/4057937618 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:26.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79b9d9b700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79bc1587f0 0x7f79bc1f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.543+0000 7f79ba59c700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 0x7f79bc153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79bad9d700 1 -- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 msgr2=0x7f79bc153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79bad9d700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 0x7f79bc153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79bad9d700 1 -- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79bc1587f0 msgr2=0x7f79bc1f7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79bad9d700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79bc1587f0 0x7f79bc1f7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79bad9d700 1 -- 172.21.15.89:0/4057937618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b000b7f0 con 0x7f79bc15b370 2023-11-24T06:19:26.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79b9d9b700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79bc1587f0 0x7f79bc1f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:26.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79bad9d700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc0aef60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f79b400cd70 tx=0x7f79b400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79aaffd700 1 -- 172.21.15.89:0/4057937618 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4003a20 con 0x7f79bc15b370 2023-11-24T06:19:26.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79aaffd700 1 -- 172.21.15.89:0/4057937618 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b4003b80 con 0x7f79bc15b370 2023-11-24T06:19:26.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79aaffd700 1 -- 172.21.15.89:0/4057937618 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4004020 con 0x7f79bc15b370 2023-11-24T06:19:26.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79bc1f40c0 con 0x7f79bc15b370 2023-11-24T06:19:26.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.544+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79bc1f4680 con 0x7f79bc15b370 2023-11-24T06:19:26.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.545+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79bc1f3f20 con 0x7f79bc15b370 2023-11-24T06:19:26.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.545+0000 7f79aaffd700 1 -- 172.21.15.89:0/4057937618 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f79b4004180 con 0x7f79bc15b370 2023-11-24T06:19:26.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.546+0000 7f79aaffd700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f799c06aed0 0x7f799c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.546+0000 7f79aaffd700 1 -- 172.21.15.89:0/4057937618 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79b4012070 con 0x7f79bc15b370 2023-11-24T06:19:26.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.546+0000 7f79ba59c700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f799c06aed0 0x7f799c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.546+0000 7f79ba59c700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f799c06aed0 0x7f799c06d390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00a910 tx=0x7f79ac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.551+0000 7f79aaffd700 1 -- 172.21.15.89:0/4057937618 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79bc1f3f20 con 0x7f79bc15b370 2023-11-24T06:19:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.743+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f79bc1f3f20 con 0x7f79bc15b370 2023-11-24T06:19:26.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.743+0000 7f79aaffd700 1 -- 172.21.15.89:0/4057937618 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f79bc1f3f20 con 0x7f79bc15b370 2023-11-24T06:19:26.745 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:26.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f799c06aed0 msgr2=0x7f799c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f799c06aed0 0x7f799c06d390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00a910 tx=0x7f79ac008040 comp rx=0 tx=0).stop 2023-11-24T06:19:26.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 msgr2=0x7f79bc0aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc0aef60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f79b400cd70 tx=0x7f79b400a570 comp rx=0 tx=0).stop 2023-11-24T06:19:26.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 shutdown_connections 2023-11-24T06:19:26.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79bc15b370 0x7f79bc0aef60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79bc0b3c60 0x7f79bc153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f799c06aed0 0x7f799c06d390 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 --2- 172.21.15.89:0/4057937618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79bc1587f0 0x7f79bc1f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 >> 172.21.15.89:0/4057937618 conn(0x7f79bc0d0bf0 msgr2=0x7f79bc15dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:26.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 shutdown_connections 2023-11-24T06:19:26.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.745+0000 7f79c0f62700 1 -- 172.21.15.89:0/4057937618 wait complete. 2023-11-24T06:19:26.760 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:26.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.880+0000 7f53538af700 1 Processor -- start 2023-11-24T06:19:26.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.880+0000 7f53538af700 1 -- start start 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f53538af700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 0x7f534c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f53538af700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f534c0afc90 0x7f534c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f53538af700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534c0b3c60 con 0x7f534c15e160 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f53538af700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f534c0b4500 con 0x7f534c0afc90 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f53538af700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0b58c0 con 0x7f534c0746a0 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f5351e4c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f534c0afc90 0x7f534c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f5351e4c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f534c0afc90 0x7f534c06d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36294/0 (socket says 172.21.15.89:36294) 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f535164b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f534c15e160 0x7f534c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39794/0 (socket says 172.21.15.89:39794) 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.881+0000 7f5351e4c700 1 -- 172.21.15.89:0/3507003789 learned_addr learned my addr 172.21.15.89:0/3507003789 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5350e4a700 1 --2- 172.21.15.89:0/3507003789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 0x7f534c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5351e4c700 1 -- 172.21.15.89:0/3507003789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 msgr2=0x7f534c0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:26.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5351e4c700 1 --2- 172.21.15.89:0/3507003789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 0x7f534c0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5351e4c700 1 -- 172.21.15.89:0/3507003789 >> v1:172.21.15.89:6789/0 conn(0x7f534c15e160 legacy=0x7f534c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5351e4c700 1 -- 172.21.15.89:0/3507003789 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c0b0140 con 0x7f534c0afc90 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5343fff700 1 -- 172.21.15.89:0/3507003789 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311142575 0 0) 0x7f534c0b3c60 con 0x7f534c15e160 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5351e4c700 1 --2- 172.21.15.89:0/3507003789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c06d930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5344009e50 tx=0x7f534400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e28a654a5451c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5343fff700 1 -- 172.21.15.89:0/3507003789 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534400e070 con 0x7f534c0afc90 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5343fff700 1 -- 172.21.15.89:0/3507003789 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534400b910 con 0x7f534c0afc90 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.882+0000 7f5343fff700 1 -- 172.21.15.89:0/3507003789 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534400bcc0 con 0x7f534c0afc90 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.883+0000 7f53538af700 1 -- 172.21.15.89:0/3507003789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 msgr2=0x7f534c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.883+0000 7f53538af700 1 --2- 172.21.15.89:0/3507003789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c06d930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5344009e50 tx=0x7f534400b330 comp rx=0 tx=0).stop 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.883+0000 7f53538af700 1 -- 172.21.15.89:0/3507003789 shutdown_connections 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.883+0000 7f53538af700 1 --2- 172.21.15.89:0/3507003789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c06d930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.883+0000 7f53538af700 1 --2- 172.21.15.89:0/3507003789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 0x7f534c0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.883+0000 7f53538af700 1 -- 172.21.15.89:0/3507003789 >> 172.21.15.89:0/3507003789 conn(0x7f534c0d0bf0 msgr2=0x7f534c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:26.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.883+0000 7f53538af700 1 -- 172.21.15.89:0/3507003789 shutdown_connections 2023-11-24T06:19:26.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.883+0000 7f53538af700 1 -- 172.21.15.89:0/3507003789 wait complete. 2023-11-24T06:19:26.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.884+0000 7f53538af700 1 Processor -- start 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.884+0000 7f53538af700 1 -- start start 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f53538af700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 0x7f534c14fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f53538af700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c1f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f53538af700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534c15e160 0x7f534c153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f53538af700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c14ffc0 con 0x7f534c0afc90 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f53538af700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0b0540 con 0x7f534c15e160 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f53538af700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0b0820 con 0x7f534c0746a0 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c1f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c1f4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36298/0 (socket says 172.21.15.89:36298) 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 -- 172.21.15.89:0/4040930189 learned_addr learned my addr 172.21.15.89:0/4040930189 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:26.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f535164b700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 0x7f534c14fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5351e4c700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534c15e160 0x7f534c153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 -- 172.21.15.89:0/4040930189 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 msgr2=0x7f534c14fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 0x7f534c14fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 -- 172.21.15.89:0/4040930189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534c15e160 msgr2=0x7f534c153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534c15e160 0x7f534c153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 -- 172.21.15.89:0/4040930189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5344009b30 con 0x7f534c0afc90 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5350e4a700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c1f4d90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f534800e9b0 tx=0x7f534800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5341ffb700 1 -- 172.21.15.89:0/4040930189 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348009710 con 0x7f534c0afc90 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5341ffb700 1 -- 172.21.15.89:0/4040930189 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534800c850 con 0x7f534c0afc90 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f5341ffb700 1 -- 172.21.15.89:0/4040930189 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534800cbc0 con 0x7f534c0afc90 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534c0b0b00 con 0x7f534c0afc90 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.885+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534c0b10c0 con 0x7f534c0afc90 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.886+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534c0b1250 con 0x7f534c0afc90 2023-11-24T06:19:26.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.887+0000 7f5341ffb700 1 -- 172.21.15.89:0/4040930189 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f534800cd20 con 0x7f534c0afc90 2023-11-24T06:19:26.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.887+0000 7f5341ffb700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f533006aed0 0x7f533006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:26.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.887+0000 7f5341ffb700 1 -- 172.21.15.89:0/4040930189 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53480896d0 con 0x7f534c0afc90 2023-11-24T06:19:26.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.887+0000 7f535164b700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f533006aed0 0x7f533006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:26.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.888+0000 7f535164b700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f533006aed0 0x7f533006d390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f533c000f60 tx=0x7f533c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:26.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:26.892+0000 7f5341ffb700 1 -- 172.21.15.89:0/4040930189 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5348054780 con 0x7f534c0afc90 2023-11-24T06:19:27.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.081+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f534c068fb0 con 0x7f534c0afc90 2023-11-24T06:19:27.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.082+0000 7f5341ffb700 1 -- 172.21.15.89:0/4040930189 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5348057da0 con 0x7f534c0afc90 2023-11-24T06:19:27.084 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f533006aed0 msgr2=0x7f533006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f533006aed0 0x7f533006d390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f533c000f60 tx=0x7f533c009040 comp rx=0 tx=0).stop 2023-11-24T06:19:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 msgr2=0x7f534c1f4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c1f4d90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f534800e9b0 tx=0x7f534800ed70 comp rx=0 tx=0).stop 2023-11-24T06:19:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 shutdown_connections 2023-11-24T06:19:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f533006aed0 0x7f533006d390 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534c15e160 0x7f534c153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f534c0746a0 0x7f534c14fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 --2- 172.21.15.89:0/4040930189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534c0afc90 0x7f534c1f4d90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 >> 172.21.15.89:0/4040930189 conn(0x7f534c0d0bf0 msgr2=0x7f534c1623f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 shutdown_connections 2023-11-24T06:19:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.084+0000 7f53538af700 1 -- 172.21.15.89:0/4040930189 wait complete. 2023-11-24T06:19:27.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:19:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.218+0000 7f2162f2f700 1 Processor -- start 2023-11-24T06:19:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.218+0000 7f2162f2f700 1 -- start start 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f2162f2f700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f215c15e080 0x7f215c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f2162f2f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 0x7f215c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f2162f2f700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c0ae430 con 0x7f215c06c000 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f2162f2f700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215c0af0b0 con 0x7f215c15e080 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f2162f2f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c0b05a0 con 0x7f215c0af4d0 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f21614cc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f215c06c000 0x7f215c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39810/0 (socket says 172.21.15.89:39810) 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f21614cc700 1 -- 172.21.15.89:0/2597096472 learned_addr learned my addr 172.21.15.89:0/2597096472 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f2160ccb700 1 --2- 172.21.15.89:0/2597096472 >> v2:172.21.15.123:3300/0 conn(0x7f215c15e080 0x7f215c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215bfff700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 0x7f215c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811098910 0 0) 0x7f215c0ae430 con 0x7f215c06c000 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f214c0035b0 con 0x7f215c06c000 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2101606975 0 0) 0x7f214c0035b0 con 0x7f215c06c000 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 msgr2=0x7f215c0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 0x7f215c0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 >> v2:172.21.15.123:3300/0 conn(0x7f215c15e080 msgr2=0x7f215c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 --2- 172.21.15.89:0/2597096472 >> v2:172.21.15.123:3300/0 conn(0x7f215c15e080 0x7f215c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c0ae710 con 0x7f215c06c000 2023-11-24T06:19:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f21540019f0 con 0x7f215c06c000 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 >> v1:172.21.15.89:6789/0 conn(0x7f215c06c000 legacy=0x7f215c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f214c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.219+0000 7f215b7fe700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f214c004850 0x7f214c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215b7fe700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 0x7f214c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c0af0b0 con 0x7f214c003a30 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c0b05a0 con 0x7f214c004850 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214c00ba10 con 0x7f215c0af4d0 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f2160ccb700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f214c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f21614cc700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 0x7f214c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215bfff700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f214c004850 0x7f214c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f2160ccb700 1 -- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 msgr2=0x7f214c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f2160ccb700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 0x7f214c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f2160ccb700 1 -- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f214c004850 msgr2=0x7f214c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f2160ccb700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f214c004850 0x7f214c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f2160ccb700 1 -- 172.21.15.89:0/2597096472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c0035b0 con 0x7f214c003a30 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f21614cc700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c0af4d0 0x7f214c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215bfff700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f214c004850 0x7f214c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f2160ccb700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f214c003e50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2150009b60 tx=0x7f214c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa088bd1260871ab server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215000e6c0 con 0x7f214c003a30 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215000e820 con 0x7f214c003a30 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.220+0000 7f215b7fe700 1 -- 172.21.15.89:0/2597096472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215000eb90 con 0x7f214c003a30 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.221+0000 7f2162f2f700 1 -- 172.21.15.89:0/2597096472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 msgr2=0x7f214c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.221+0000 7f2162f2f700 1 --2- 172.21.15.89:0/2597096472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f214c003e50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2150009b60 tx=0x7f214c005880 comp rx=0 tx=0).stop 2023-11-24T06:19:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.221+0000 7f21614cc700 1 -- 172.21.15.89:0/2597096472 reap_dead start 2023-11-24T06:19:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.221+0000 7f2162f2f700 1 -- 172.21.15.89:0/2597096472 shutdown_connections 2023-11-24T06:19:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.221+0000 7f2162f2f700 1 -- 172.21.15.89:0/2597096472 >> 172.21.15.89:0/2597096472 conn(0x7f215c071b90 msgr2=0x7f215c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.221+0000 7f2162f2f700 1 -- 172.21.15.89:0/2597096472 shutdown_connections 2023-11-24T06:19:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.221+0000 7f2162f2f700 1 -- 172.21.15.89:0/2597096472 wait complete. 2023-11-24T06:19:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.222+0000 7f2162f2f700 1 Processor -- start 2023-11-24T06:19:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.222+0000 7f2162f2f700 1 -- start start 2023-11-24T06:19:27.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2162f2f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f215c15ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2162f2f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f215c15f1b0 0x7f215c1e8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2162f2f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c1e8a80 0x7f215c1e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2162f2f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c1e9440 con 0x7f214c003a30 2023-11-24T06:19:27.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2162f2f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c15f980 con 0x7f215c15f1b0 2023-11-24T06:19:27.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2162f2f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c15fc60 con 0x7f215c1e8a80 2023-11-24T06:19:27.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f215c15ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f215c15ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36316/0 (socket says 172.21.15.89:36316) 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 -- 172.21.15.89:0/2497661484 learned_addr learned my addr 172.21.15.89:0/2497661484 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f215bfff700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f215c15f1b0 0x7f215c1e8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f21614cc700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c1e8a80 0x7f215c1e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 -- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c1e8a80 msgr2=0x7f215c1e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c1e8a80 0x7f215c1e8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 -- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f215c15f1b0 msgr2=0x7f215c1e8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f215c15f1b0 0x7f215c1e8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 -- 172.21.15.89:0/2497661484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c15ff40 con 0x7f214c003a30 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f21614cc700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c1e8a80 0x7f215c1e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2160ccb700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f215c15ec70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f21500090b0 tx=0x7f21500047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:27.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f21597fa700 1 -- 172.21.15.89:0/2497661484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215000e6c0 con 0x7f214c003a30 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f21597fa700 1 -- 172.21.15.89:0/2497661484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2150003ad0 con 0x7f214c003a30 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c1e9440 con 0x7f214c003a30 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.223+0000 7f21597fa700 1 -- 172.21.15.89:0/2497661484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150003e80 con 0x7f214c003a30 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.224+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c160500 con 0x7f214c003a30 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.224+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c0753f0 con 0x7f214c003a30 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.225+0000 7f21597fa700 1 -- 172.21.15.89:0/2497661484 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2150013470 con 0x7f214c003a30 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.225+0000 7f21597fa700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f214006aed0 0x7f214006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.225+0000 7f21597fa700 1 -- 172.21.15.89:0/2497661484 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f215008aff0 con 0x7f214c003a30 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.225+0000 7f215bfff700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f214006aed0 0x7f214006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.226+0000 7f215bfff700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f214006aed0 0x7f214006d390 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f215c14e360 tx=0x7f2148008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:27.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.230+0000 7f21597fa700 1 -- 172.21.15.89:0/2497661484 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2150056210 con 0x7f214c003a30 2023-11-24T06:19:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.435+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f215c160690 con 0x7f214c003a30 2023-11-24T06:19:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.435+0000 7f21597fa700 1 -- 172.21.15.89:0/2497661484 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2150059830 con 0x7f214c003a30 2023-11-24T06:19:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f214006aed0 msgr2=0x7f214006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f214006aed0 0x7f214006d390 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f215c14e360 tx=0x7f2148008040 comp rx=0 tx=0).stop 2023-11-24T06:19:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 msgr2=0x7f215c15ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f215c15ec70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f21500090b0 tx=0x7f21500047c0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 shutdown_connections 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f215c1e8a80 0x7f215c1e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c003a30 0x7f215c15ec70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f214006aed0 0x7f214006d390 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 --2- 172.21.15.89:0/2497661484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f215c15f1b0 0x7f215c1e8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 >> 172.21.15.89:0/2497661484 conn(0x7f215c071b90 msgr2=0x7f215c159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 shutdown_connections 2023-11-24T06:19:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.437+0000 7f2162f2f700 1 -- 172.21.15.89:0/2497661484 wait complete. 2023-11-24T06:19:27.441 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2023-11-24T06:19:27.451 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-11-24T06:19:27.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:27.452 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-11-24T06:19:27.487 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:19:27.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:27.488 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-24T06:19:27.544 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:27.545 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:27.566 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:27.566 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:27.626 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:27.626 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:27.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:27.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.806+0000 7f6bf0f77700 1 Processor -- start 2023-11-24T06:19:27.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.806+0000 7f6bf0f77700 1 -- start start 2023-11-24T06:19:27.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bf0f77700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6bec15e080 0x7f6bec0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bf0f77700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bec0ae430 con 0x7f6bec06c710 2023-11-24T06:19:27.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bf0f77700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bec0ae710 con 0x7f6bec0af4d0 2023-11-24T06:19:27.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bf0f77700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bec0af0b0 con 0x7f6bec15e080 2023-11-24T06:19:27.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bea59c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6bec15e080 0x7f6bec0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bead9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6bec06c710 0x7f6bec06e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39832/0 (socket says 172.21.15.89:39832) 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bead9d700 1 -- 172.21.15.89:0/278198902 learned_addr learned my addr 172.21.15.89:0/278198902 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bea59c700 1 -- 172.21.15.89:0/278198902 >> v1:172.21.15.123:6789/0 conn(0x7f6bec0af4d0 legacy=0x7f6bec06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bea59c700 1 -- 172.21.15.89:0/278198902 >> v1:172.21.15.89:6789/0 conn(0x7f6bec06c710 legacy=0x7f6bec06e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6be959a700 1 -- 172.21.15.89:0/278198902 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144378559 0 0) 0x7f6bec0ae430 con 0x7f6bec06c710 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bea59c700 1 -- 172.21.15.89:0/278198902 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec0ae9f0 con 0x7f6bec15e080 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6bea59c700 1 --2- 172.21.15.89:0/278198902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec15e080 0x7f6bec0b5050 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009840 tx=0x7f6bdc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3891c5f8727220a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6be959a700 1 -- 172.21.15.89:0/278198902 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc011830 con 0x7f6bec15e080 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6be959a700 1 -- 172.21.15.89:0/278198902 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc011990 con 0x7f6bec15e080 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.807+0000 7f6be959a700 1 -- 172.21.15.89:0/278198902 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc011d20 con 0x7f6bec15e080 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.808+0000 7f6bf0f77700 1 -- 172.21.15.89:0/278198902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec15e080 msgr2=0x7f6bec0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.808+0000 7f6bf0f77700 1 --2- 172.21.15.89:0/278198902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec15e080 0x7f6bec0b5050 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009840 tx=0x7f6bdc009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:27.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.808+0000 7f6bf0f77700 1 -- 172.21.15.89:0/278198902 shutdown_connections 2023-11-24T06:19:27.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.808+0000 7f6bf0f77700 1 --2- 172.21.15.89:0/278198902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec15e080 0x7f6bec0b5050 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.808+0000 7f6bf0f77700 1 -- 172.21.15.89:0/278198902 >> 172.21.15.89:0/278198902 conn(0x7f6bec071b90 msgr2=0x7f6bec071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:27.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.808+0000 7f6bf0f77700 1 -- 172.21.15.89:0/278198902 shutdown_connections 2023-11-24T06:19:27.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.808+0000 7f6bf0f77700 1 -- 172.21.15.89:0/278198902 wait complete. 2023-11-24T06:19:27.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.809+0000 7f6bf0f77700 1 Processor -- start 2023-11-24T06:19:27.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.809+0000 7f6bf0f77700 1 -- start start 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bf0f77700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec06c710 0x7f6bec146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bf0f77700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bec0af4d0 0x7f6bec15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bf0f77700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bec15e080 0x7f6bec159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bf0f77700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec158a60 con 0x7f6bec06c710 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bf0f77700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec1584a0 con 0x7f6bec0af4d0 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bf0f77700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec158780 con 0x7f6bec15e080 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bea59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec06c710 0x7f6bec146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bea59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec06c710 0x7f6bec146550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36330/0 (socket says 172.21.15.89:36330) 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6bea59c700 1 -- 172.21.15.89:0/3009784159 learned_addr learned my addr 172.21.15.89:0/3009784159 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.810+0000 7f6be9d9b700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bec0af4d0 0x7f6bec15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bead9d700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bec15e080 0x7f6bec159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bea59c700 1 -- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bec15e080 msgr2=0x7f6bec159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bea59c700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bec15e080 0x7f6bec159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bea59c700 1 -- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bec0af4d0 msgr2=0x7f6bec15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bea59c700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bec0af4d0 0x7f6bec15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bea59c700 1 -- 172.21.15.89:0/3009784159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0094a0 con 0x7f6bec06c710 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bead9d700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bec15e080 0x7f6bec159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bea59c700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec06c710 0x7f6bec146550 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0090b0 tx=0x7f6bdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bdaffd700 1 -- 172.21.15.89:0/3009784159 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc0036a0 con 0x7f6bec06c710 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bdaffd700 1 -- 172.21.15.89:0/3009784159 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc01e070 con 0x7f6bec06c710 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bdaffd700 1 -- 172.21.15.89:0/3009784159 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc003a20 con 0x7f6bec06c710 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bec15e590 con 0x7f6bec06c710 2023-11-24T06:19:27.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.811+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bec146a90 con 0x7f6bec06c710 2023-11-24T06:19:27.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.812+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bec0753f0 con 0x7f6bec06c710 2023-11-24T06:19:27.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.812+0000 7f6bdaffd700 1 -- 172.21.15.89:0/3009784159 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6bdc003b80 con 0x7f6bec06c710 2023-11-24T06:19:27.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.813+0000 7f6bdaffd700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6bcc06aed0 0x7f6bcc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:27.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.813+0000 7f6bdaffd700 1 -- 172.21.15.89:0/3009784159 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bdc08a6e0 con 0x7f6bec06c710 2023-11-24T06:19:27.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.813+0000 7f6be9d9b700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6bcc06aed0 0x7f6bcc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:27.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.813+0000 7f6be9d9b700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6bcc06aed0 0x7f6bcc06d390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6bec156dd0 tx=0x7f6be0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:27.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:27.817+0000 7f6bdaffd700 1 -- 172.21.15.89:0/3009784159 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6bdc055820 con 0x7f6bec06c710 2023-11-24T06:19:28.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.007+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bec15eab0 con 0x7f6bec06c710 2023-11-24T06:19:28.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.009+0000 7f6bdaffd700 1 -- 172.21.15.89:0/3009784159 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f6bec15eab0 con 0x7f6bec06c710 2023-11-24T06:19:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:28.011 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:18:51.014670+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:28.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6bcc06aed0 msgr2=0x7f6bcc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6bcc06aed0 0x7f6bcc06d390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6bec156dd0 tx=0x7f6be0009040 comp rx=0 tx=0).stop 2023-11-24T06:19:28.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec06c710 msgr2=0x7f6bec146550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec06c710 0x7f6bec146550 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0090b0 tx=0x7f6bdc004750 comp rx=0 tx=0).stop 2023-11-24T06:19:28.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 shutdown_connections 2023-11-24T06:19:28.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bec15e080 0x7f6bec159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec06c710 0x7f6bec146550 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6bcc06aed0 0x7f6bcc06d390 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 --2- 172.21.15.89:0/3009784159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bec0af4d0 0x7f6bec15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 >> 172.21.15.89:0/3009784159 conn(0x7f6bec071b90 msgr2=0x7f6bec0b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:28.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 shutdown_connections 2023-11-24T06:19:28.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.011+0000 7f6bf0f77700 1 -- 172.21.15.89:0/3009784159 wait complete. 2023-11-24T06:19:28.025 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:28.025 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.146+0000 7fc4b2476700 1 Processor -- start 2023-11-24T06:19:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.146+0000 7fc4b2476700 1 -- start start 2023-11-24T06:19:28.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4b2476700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4ac0cfdf0 0x7fc4ac073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4b2476700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac0b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4b2476700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac0b5ee0 con 0x7fc4ac15be70 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4b2476700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac06bfc0 con 0x7fc4ac0cfdf0 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4b2476700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac074960 con 0x7fc4ac0744d0 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4ab7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4ac0cfdf0 0x7fc4ac073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4abfff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4ac15be70 0x7fc4ac0add40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39842/0 (socket says 172.21.15.89:39842) 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4abfff700 1 -- 172.21.15.89:0/19092481 learned_addr learned my addr 172.21.15.89:0/19092481 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4b0a13700 1 --2- 172.21.15.89:0/19092481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac0b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4ab7fe700 1 -- 172.21.15.89:0/19092481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 msgr2=0x7fc4ac0b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4ab7fe700 1 --2- 172.21.15.89:0/19092481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac0b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4ab7fe700 1 -- 172.21.15.89:0/19092481 >> v1:172.21.15.89:6789/0 conn(0x7fc4ac15be70 legacy=0x7fc4ac0add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4ab7fe700 1 -- 172.21.15.89:0/19092481 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac06ba00 con 0x7fc4ac0cfdf0 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4b0a13700 1 --2- 172.21.15.89:0/19092481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac0b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.147+0000 7fc4ab7fe700 1 --2- 172.21.15.89:0/19092481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac0cfdf0 0x7fc4ac073f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009840 tx=0x7fc4a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9d6760fd6951a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4aaffd700 1 -- 172.21.15.89:0/19092481 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0011830 con 0x7fc4ac0cfdf0 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4aaffd700 1 -- 172.21.15.89:0/19092481 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a0011990 con 0x7fc4ac0cfdf0 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4aaffd700 1 -- 172.21.15.89:0/19092481 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0011d20 con 0x7fc4ac0cfdf0 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4b2476700 1 -- 172.21.15.89:0/19092481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac0cfdf0 msgr2=0x7fc4ac073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4b2476700 1 --2- 172.21.15.89:0/19092481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac0cfdf0 0x7fc4ac073f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009840 tx=0x7fc4a0009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4b2476700 1 -- 172.21.15.89:0/19092481 shutdown_connections 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4b2476700 1 --2- 172.21.15.89:0/19092481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac0b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4b2476700 1 --2- 172.21.15.89:0/19092481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac0cfdf0 0x7fc4ac073f90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4b2476700 1 -- 172.21.15.89:0/19092481 >> 172.21.15.89:0/19092481 conn(0x7fc4ac0d0bf0 msgr2=0x7fc4ac0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4b2476700 1 -- 172.21.15.89:0/19092481 shutdown_connections 2023-11-24T06:19:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.148+0000 7fc4b2476700 1 -- 172.21.15.89:0/19092481 wait complete. 2023-11-24T06:19:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.150+0000 7fc4b2476700 1 Processor -- start 2023-11-24T06:19:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.150+0000 7fc4b2476700 1 -- start start 2023-11-24T06:19:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b2476700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac15e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b2476700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4ac0cfdf0 0x7fc4ac150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b2476700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac15be70 0x7fc4ac1508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b2476700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac09ab10 con 0x7fc4ac15be70 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b2476700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac150df0 con 0x7fc4ac0cfdf0 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b2476700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac15d100 con 0x7fc4ac0744d0 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac15be70 0x7fc4ac1508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac15be70 0x7fc4ac1508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36348/0 (socket says 172.21.15.89:36348) 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 -- 172.21.15.89:0/3080862428 learned_addr learned my addr 172.21.15.89:0/3080862428 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4abfff700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac15e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4ab7fe700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4ac0cfdf0 0x7fc4ac150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 -- 172.21.15.89:0/3080862428 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 msgr2=0x7fc4ac15e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac15e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 -- 172.21.15.89:0/3080862428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4ac0cfdf0 msgr2=0x7fc4ac150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4ac0cfdf0 0x7fc4ac150370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 -- 172.21.15.89:0/3080862428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a00094a0 con 0x7fc4ac15be70 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b0a13700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac15be70 0x7fc4ac1508b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400cef0 tx=0x7fc4a400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4a8ff9700 1 -- 172.21.15.89:0/3080862428 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400ad00 con 0x7fc4ac15be70 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4a8ff9700 1 -- 172.21.15.89:0/3080862428 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a400ae60 con 0x7fc4ac15be70 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4a8ff9700 1 -- 172.21.15.89:0/3080862428 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4015680 con 0x7fc4ac15be70 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.151+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac15d3e0 con 0x7fc4ac15be70 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.152+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac15d9a0 con 0x7fc4ac15be70 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.152+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac068fb0 con 0x7fc4ac15be70 2023-11-24T06:19:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.153+0000 7fc4a8ff9700 1 -- 172.21.15.89:0/3080862428 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc4a400d430 con 0x7fc4ac15be70 2023-11-24T06:19:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.153+0000 7fc4a8ff9700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc49806aed0 0x7fc49806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.153+0000 7fc4a8ff9700 1 -- 172.21.15.89:0/3080862428 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4a4089ab0 con 0x7fc4ac15be70 2023-11-24T06:19:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.153+0000 7fc4abfff700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc49806aed0 0x7fc49806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.154+0000 7fc4abfff700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc49806aed0 0x7fc49806d390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc49c005910 tx=0x7fc49c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.158+0000 7fc4a8ff9700 1 -- 172.21.15.89:0/3080862428 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4a4054b50 con 0x7fc4ac15be70 2023-11-24T06:19:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.347+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4ac0b26d0 con 0x7fc4ac15be70 2023-11-24T06:19:28.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.348+0000 7fc4a8ff9700 1 -- 172.21.15.89:0/3080862428 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4a4058170 con 0x7fc4ac15be70 2023-11-24T06:19:28.350 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:28.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc49806aed0 msgr2=0x7fc49806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc49806aed0 0x7fc49806d390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc49c005910 tx=0x7fc49c009040 comp rx=0 tx=0).stop 2023-11-24T06:19:28.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac15be70 msgr2=0x7fc4ac1508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac15be70 0x7fc4ac1508b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400cef0 tx=0x7fc4a400cf20 comp rx=0 tx=0).stop 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 shutdown_connections 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ac15be70 0x7fc4ac1508b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4ac0744d0 0x7fc4ac15e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc49806aed0 0x7fc49806d390 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 --2- 172.21.15.89:0/3080862428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4ac0cfdf0 0x7fc4ac150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 >> 172.21.15.89:0/3080862428 conn(0x7fc4ac0d0bf0 msgr2=0x7fc4ac0b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 shutdown_connections 2023-11-24T06:19:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.350+0000 7fc4b2476700 1 -- 172.21.15.89:0/3080862428 wait complete. 2023-11-24T06:19:28.364 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.485+0000 7f725f217700 1 Processor -- start 2023-11-24T06:19:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.485+0000 7f725f217700 1 -- start start 2023-11-24T06:19:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725f217700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f725815f9e0 0x7f72580add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725f217700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7258073f90 0x7f7258074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725f217700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f725806b080 con 0x7f72580cf610 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725f217700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7258074960 con 0x7f725815f9e0 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725f217700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725815c290 con 0x7f7258073f90 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725cfb3700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f72580cf610 0x7f72580b3c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39856/0 (socket says 172.21.15.89:39856) 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725cfb3700 1 -- 172.21.15.89:0/3007625019 learned_addr learned my addr 172.21.15.89:0/3007625019 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263766047 0 0) 0x7f725806b080 con 0x7f72580cf610 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f7257fff700 1 --2- 172.21.15.89:0/3007625019 >> v2:172.21.15.123:3300/0 conn(0x7f725815f9e0 0x7f72580add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72480035b0 con 0x7f72580cf610 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725d7b4700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7258073f90 0x7f7258074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523849273 0 0) 0x7f72480035b0 con 0x7f72580cf610 2023-11-24T06:19:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7258073f90 msgr2=0x7f7258074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7258073f90 0x7f7258074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 >> v2:172.21.15.123:3300/0 conn(0x7f725815f9e0 msgr2=0x7f72580add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 --2- 172.21.15.89:0/3007625019 >> v2:172.21.15.123:3300/0 conn(0x7f725815f9e0 0x7f72580add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580cf9f0 con 0x7f72580cf610 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f724c001670 con 0x7f72580cf610 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 >> v1:172.21.15.89:6789/0 conn(0x7f72580cf610 legacy=0x7f72580b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7248003a00 0x7f7248003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7258073f90 0x7f7248004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248004d90 0x7f7248005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7258074960 con 0x7f7248003a00 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725815c290 con 0x7f7248004d90 2023-11-24T06:19:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724800ba10 con 0x7f7258073f90 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.486+0000 7f725cfb3700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7248003a00 0x7f7248003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f7257fff700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7258073f90 0x7f7248004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725d7b4700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248004d90 0x7f7248005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725cfb3700 1 -- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7258073f90 msgr2=0x7f7248004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725cfb3700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7258073f90 0x7f7248004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725cfb3700 1 -- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248004d90 msgr2=0x7f7248005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725cfb3700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248004d90 0x7f7248005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725cfb3700 1 -- 172.21.15.89:0/3007625019 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72480035b0 con 0x7f7248003a00 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725d7b4700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248004d90 0x7f7248005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725cfb3700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7248003a00 0x7f7248003e20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f724c00c3a0 tx=0x7f7248005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5974a9348e77e89 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c007c60 con 0x7f7248003a00 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724c005960 con 0x7f7248003a00 2023-11-24T06:19:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f72577fe700 1 -- 172.21.15.89:0/3007625019 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c00cea0 con 0x7f7248003a00 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725f217700 1 -- 172.21.15.89:0/3007625019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7248003a00 msgr2=0x7f7248003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725f217700 1 --2- 172.21.15.89:0/3007625019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7248003a00 0x7f7248003e20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f724c00c3a0 tx=0x7f7248005880 comp rx=0 tx=0).stop 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.487+0000 7f725d7b4700 1 -- 172.21.15.89:0/3007625019 reap_dead start 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.488+0000 7f725f217700 1 -- 172.21.15.89:0/3007625019 shutdown_connections 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.488+0000 7f725f217700 1 -- 172.21.15.89:0/3007625019 >> 172.21.15.89:0/3007625019 conn(0x7f72580d0bf0 msgr2=0x7f72580d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.488+0000 7f725f217700 1 -- 172.21.15.89:0/3007625019 shutdown_connections 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.488+0000 7f725f217700 1 -- 172.21.15.89:0/3007625019 wait complete. 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.489+0000 7f725f217700 1 Processor -- start 2023-11-24T06:19:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.489+0000 7f725f217700 1 -- start start 2023-11-24T06:19:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725f217700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248003a00 0x7f725814d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725f217700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f725809c6d0 0x7f725809ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725f217700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725809b0d0 0x7f725815e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725f217700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725814df30 con 0x7f725809b0d0 2023-11-24T06:19:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725f217700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725809b630 con 0x7f7248003a00 2023-11-24T06:19:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725f217700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725809b910 con 0x7f725809c6d0 2023-11-24T06:19:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725809b0d0 0x7f725815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725809b0d0 0x7f725815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36368/0 (socket says 172.21.15.89:36368) 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 -- 172.21.15.89:0/324583814 learned_addr learned my addr 172.21.15.89:0/324583814 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f7257fff700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f725809c6d0 0x7f725809ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725cfb3700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248003a00 0x7f725814d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 -- 172.21.15.89:0/324583814 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f725809c6d0 msgr2=0x7f725809ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f725809c6d0 0x7f725809ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 -- 172.21.15.89:0/324583814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248003a00 msgr2=0x7f725814d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248003a00 0x7f725814d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 -- 172.21.15.89:0/324583814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725809bbf0 con 0x7f725809b0d0 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725d7b4700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725809b0d0 0x7f725815e680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7258072c60 tx=0x7f725000f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f72557fa700 1 -- 172.21.15.89:0/324583814 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725000fd60 con 0x7f725809b0d0 2023-11-24T06:19:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f72557fa700 1 -- 172.21.15.89:0/324583814 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7250011040 con 0x7f725809b0d0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725809bed0 con 0x7f725809b0d0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.490+0000 7f72557fa700 1 -- 172.21.15.89:0/324583814 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725000fee0 con 0x7f725809b0d0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.491+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7258159250 con 0x7f725809b0d0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.491+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72580ae720 con 0x7f725809b0d0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.492+0000 7f72557fa700 1 -- 172.21.15.89:0/324583814 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f72500203b0 con 0x7f725809b0d0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.492+0000 7f72557fa700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f723c06aed0 0x7f723c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.492+0000 7f72557fa700 1 -- 172.21.15.89:0/324583814 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72500896b0 con 0x7f725809b0d0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.492+0000 7f725cfb3700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f723c06aed0 0x7f723c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.493+0000 7f725cfb3700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f723c06aed0 0x7f723c06d390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f724c001050 tx=0x7f724c00a000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.497+0000 7f72557fa700 1 -- 172.21.15.89:0/324583814 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7250054770 con 0x7f725809b0d0 2023-11-24T06:19:28.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.686+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7258068fb0 con 0x7f725809b0d0 2023-11-24T06:19:28.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.687+0000 7f72557fa700 1 -- 172.21.15.89:0/324583814 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7250057d90 con 0x7f725809b0d0 2023-11-24T06:19:28.689 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f723c06aed0 msgr2=0x7f723c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f723c06aed0 0x7f723c06d390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f724c001050 tx=0x7f724c00a000 comp rx=0 tx=0).stop 2023-11-24T06:19:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725809b0d0 msgr2=0x7f725815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725809b0d0 0x7f725815e680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7258072c60 tx=0x7f725000f3b0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 shutdown_connections 2023-11-24T06:19:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725809b0d0 0x7f725815e680 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f723c06aed0 0x7f723c06d390 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7248003a00 0x7f725814d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 --2- 172.21.15.89:0/324583814 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f725809c6d0 0x7f725809ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 >> 172.21.15.89:0/324583814 conn(0x7f72580d0bf0 msgr2=0x7f72580b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 shutdown_connections 2023-11-24T06:19:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.689+0000 7f725f217700 1 -- 172.21.15.89:0/324583814 wait complete. 2023-11-24T06:19:28.703 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:28.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.824+0000 7fa3baa3d700 1 Processor -- start 2023-11-24T06:19:28.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.824+0000 7fa3baa3d700 1 -- start start 2023-11-24T06:19:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3baa3d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa3b40746a0 0x7fa3b40b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3baa3d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40afc90 0x7fa3b406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3baa3d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b40b3c60 con 0x7fa3b415e160 2023-11-24T06:19:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3baa3d700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b40b4500 con 0x7fa3b40746a0 2023-11-24T06:19:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3baa3d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b40b58c0 con 0x7fa3b40afc90 2023-11-24T06:19:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b37fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa3b40746a0 0x7fa3b40b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b3fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa3b415e160 0x7fa3b4073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39876/0 (socket says 172.21.15.89:39876) 2023-11-24T06:19:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b3fff700 1 -- 172.21.15.89:0/2765560003 learned_addr learned my addr 172.21.15.89:0/2765560003 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b8fda700 1 --2- 172.21.15.89:0/2765560003 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40afc90 0x7fa3b406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b37fe700 1 -- 172.21.15.89:0/2765560003 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40afc90 msgr2=0x7fa3b406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b37fe700 1 --2- 172.21.15.89:0/2765560003 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40afc90 0x7fa3b406d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b37fe700 1 -- 172.21.15.89:0/2765560003 >> v1:172.21.15.89:6789/0 conn(0x7fa3b415e160 legacy=0x7fa3b4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b37fe700 1 -- 172.21.15.89:0/2765560003 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b40b0120 con 0x7fa3b40746a0 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b2ffd700 1 -- 172.21.15.89:0/2765560003 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522605267 0 0) 0x7fa3b40b3c60 con 0x7fa3b415e160 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.825+0000 7fa3b8fda700 1 --2- 172.21.15.89:0/2765560003 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40afc90 0x7fa3b406d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3b37fe700 1 --2- 172.21.15.89:0/2765560003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40746a0 0x7fa3b40b5da0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009840 tx=0x7fa3a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217c023aa2fd3a0e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3b2ffd700 1 -- 172.21.15.89:0/2765560003 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4011830 con 0x7fa3b40746a0 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3b2ffd700 1 -- 172.21.15.89:0/2765560003 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a4011990 con 0x7fa3b40746a0 2023-11-24T06:19:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3b2ffd700 1 -- 172.21.15.89:0/2765560003 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4011d20 con 0x7fa3b40746a0 2023-11-24T06:19:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3baa3d700 1 -- 172.21.15.89:0/2765560003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40746a0 msgr2=0x7fa3b40b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/2765560003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40746a0 0x7fa3b40b5da0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009840 tx=0x7fa3a4009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3baa3d700 1 -- 172.21.15.89:0/2765560003 shutdown_connections 2023-11-24T06:19:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/2765560003 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40afc90 0x7fa3b406d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/2765560003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40746a0 0x7fa3b40b5da0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.826+0000 7fa3baa3d700 1 -- 172.21.15.89:0/2765560003 >> 172.21.15.89:0/2765560003 conn(0x7fa3b40d0bf0 msgr2=0x7fa3b40d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.827+0000 7fa3baa3d700 1 -- 172.21.15.89:0/2765560003 shutdown_connections 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.827+0000 7fa3baa3d700 1 -- 172.21.15.89:0/2765560003 wait complete. 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.828+0000 7fa3baa3d700 1 Processor -- start 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.828+0000 7fa3baa3d700 1 -- start start 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3baa3d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40746a0 0x7fa3b4142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3baa3d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40afc90 0x7fa3b4158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3baa3d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b415e160 0x7fa3b40ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3baa3d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b4159360 con 0x7fa3b40afc90 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3baa3d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b415ec60 con 0x7fa3b415e160 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3baa3d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b4142b60 con 0x7fa3b40746a0 2023-11-24T06:19:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40afc90 0x7fa3b4158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40afc90 0x7fa3b4158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36378/0 (socket says 172.21.15.89:36378) 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 -- 172.21.15.89:0/3827744058 learned_addr learned my addr 172.21.15.89:0/3827744058 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b3fff700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40746a0 0x7fa3b4142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b8fda700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b415e160 0x7fa3b40ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 -- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40746a0 msgr2=0x7fa3b4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40746a0 0x7fa3b4142620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 -- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b415e160 msgr2=0x7fa3b40ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b415e160 0x7fa3b40ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 -- 172.21.15.89:0/3827744058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a40094a0 con 0x7fa3b40afc90 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b3fff700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40746a0 0x7fa3b4142620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b8fda700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b415e160 0x7fa3b40ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b37fe700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40afc90 0x7fa3b4158470 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4072ed0 tx=0x7fa3a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b0ff9700 1 -- 172.21.15.89:0/3827744058 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40036a0 con 0x7fa3b40afc90 2023-11-24T06:19:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b0ff9700 1 -- 172.21.15.89:0/3827744058 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a401f070 con 0x7fa3b40afc90 2023-11-24T06:19:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b4158da0 con 0x7fa3b40afc90 2023-11-24T06:19:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3b0ff9700 1 -- 172.21.15.89:0/3827744058 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4003a40 con 0x7fa3b40afc90 2023-11-24T06:19:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.829+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b415e8f0 con 0x7fa3b40afc90 2023-11-24T06:19:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.830+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b415e750 con 0x7fa3b40afc90 2023-11-24T06:19:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.831+0000 7fa3b0ff9700 1 -- 172.21.15.89:0/3827744058 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa3a4003ba0 con 0x7fa3b40afc90 2023-11-24T06:19:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.831+0000 7fa3b0ff9700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa39406aed0 0x7fa39406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.831+0000 7fa3b0ff9700 1 -- 172.21.15.89:0/3827744058 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3a4021070 con 0x7fa3b40afc90 2023-11-24T06:19:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.831+0000 7fa3b3fff700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa39406aed0 0x7fa39406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.831+0000 7fa3b3fff700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa39406aed0 0x7fa39406d390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa39c000f60 tx=0x7fa39c007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:28.835+0000 7fa3b0ff9700 1 -- 172.21.15.89:0/3827744058 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3b415e750 con 0x7fa3b40afc90 2023-11-24T06:19:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.026+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3b415ee10 con 0x7fa3b40afc90 2023-11-24T06:19:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.027+0000 7fa3b0ff9700 1 -- 172.21.15.89:0/3827744058 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3b415ee10 con 0x7fa3b40afc90 2023-11-24T06:19:29.029 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:29.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa39406aed0 msgr2=0x7fa39406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa39406aed0 0x7fa39406d390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa39c000f60 tx=0x7fa39c007d30 comp rx=0 tx=0).stop 2023-11-24T06:19:29.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40afc90 msgr2=0x7fa3b4158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40afc90 0x7fa3b4158470 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4072ed0 tx=0x7fa3a4004750 comp rx=0 tx=0).stop 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 shutdown_connections 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa39406aed0 0x7fa39406d390 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b415e160 0x7fa3b40ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3b40746a0 0x7fa3b4142620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 --2- 172.21.15.89:0/3827744058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3b40afc90 0x7fa3b4158470 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 >> 172.21.15.89:0/3827744058 conn(0x7fa3b40d0bf0 msgr2=0x7fa3b406ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 shutdown_connections 2023-11-24T06:19:29.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.029+0000 7fa3baa3d700 1 -- 172.21.15.89:0/3827744058 wait complete. 2023-11-24T06:19:29.044 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:19:29.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.163+0000 7ff788f8c700 1 Processor -- start 2023-11-24T06:19:29.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.163+0000 7ff788f8c700 1 -- start start 2023-11-24T06:19:29.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff788f8c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 0x7ff7840b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff788f8c700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7840b0290 con 0x7ff7840b4180 2023-11-24T06:19:29.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff788f8c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7840b4820 con 0x7ff7840b59f0 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff788f8c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840b4610 con 0x7ff7840b5dd0 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff782d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff7840b4180 0x7ff7840af190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39892/0 (socket says 172.21.15.89:39892) 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff782d9d700 1 -- 172.21.15.89:0/1706060913 learned_addr learned my addr 172.21.15.89:0/1706060913 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115443924 0 0) 0x7ff7840b0290 con 0x7ff7840b4180 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff76c0035b0 con 0x7ff7840b4180 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff781d9b700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 0x7ff7840b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1295665091 0 0) 0x7ff76c0035b0 con 0x7ff7840b4180 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 msgr2=0x7ff7840b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff78159a700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 0x7ff7840b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 >> v1:172.21.15.123:6789/0 conn(0x7ff7840b59f0 legacy=0x7ff78406db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7840b4b00 con 0x7ff7840b4180 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7ff77c0019f0 con 0x7ff7840b4180 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.164+0000 7ff781d9b700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 0x7ff7840b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 >> v1:172.21.15.89:6789/0 conn(0x7ff7840b4180 legacy=0x7ff7840af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 0x7ff76c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff76c004020 0x7ff76c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004d90 0x7ff76c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840b4610 con 0x7ff76c004020 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c00bb20 con 0x7ff76c004d90 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c00bce0 con 0x7ff7840b5dd0 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff781d9b700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff76c004020 0x7ff76c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78259c700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 0x7ff76c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff782d9d700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004d90 0x7ff76c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff781d9b700 1 -- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 msgr2=0x7ff76c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff781d9b700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840b5dd0 0x7ff76c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff781d9b700 1 -- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004d90 msgr2=0x7ff76c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff781d9b700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004d90 0x7ff76c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff781d9b700 1 -- 172.21.15.89:0/1706060913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0035b0 con 0x7ff76c004020 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff781d9b700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff76c004020 0x7ff76c004850 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff7840b6280 tx=0x7ff76c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae189579e79b6309 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77800c070 con 0x7ff76c004020 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff778004020 con 0x7ff76c004020 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.165+0000 7ff78159a700 1 -- 172.21.15.89:0/1706060913 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778009520 con 0x7ff76c004020 2023-11-24T06:19:29.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.166+0000 7ff788f8c700 1 -- 172.21.15.89:0/1706060913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff76c004020 msgr2=0x7ff76c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.166+0000 7ff788f8c700 1 --2- 172.21.15.89:0/1706060913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff76c004020 0x7ff76c004850 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff7840b6280 tx=0x7ff76c005880 comp rx=0 tx=0).stop 2023-11-24T06:19:29.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.166+0000 7ff782d9d700 1 -- 172.21.15.89:0/1706060913 reap_dead start 2023-11-24T06:19:29.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.166+0000 7ff788f8c700 1 -- 172.21.15.89:0/1706060913 shutdown_connections 2023-11-24T06:19:29.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.166+0000 7ff788f8c700 1 -- 172.21.15.89:0/1706060913 >> 172.21.15.89:0/1706060913 conn(0x7ff784072990 msgr2=0x7ff78406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:29.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.166+0000 7ff788f8c700 1 -- 172.21.15.89:0/1706060913 shutdown_connections 2023-11-24T06:19:29.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.166+0000 7ff788f8c700 1 -- 172.21.15.89:0/1706060913 wait complete. 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.167+0000 7ff788f8c700 1 Processor -- start 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.167+0000 7ff788f8c700 1 -- start start 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff788f8c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004020 0x7ff784149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff788f8c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff78415e8a0 0x7ff7840cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff788f8c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840cf7f0 0x7ff7840cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff788f8c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840cfc70 con 0x7ff78415e8a0 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff788f8c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7841f3a50 con 0x7ff76c004020 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff788f8c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78415a7e0 con 0x7ff7840cf7f0 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff781d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff78415e8a0 0x7ff7840cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff781d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff78415e8a0 0x7ff7840cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36398/0 (socket says 172.21.15.89:36398) 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff781d9b700 1 -- 172.21.15.89:0/586784968 learned_addr learned my addr 172.21.15.89:0/586784968 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff78259c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004020 0x7ff784149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff782d9d700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840cf7f0 0x7ff7840cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff781d9b700 1 -- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840cf7f0 msgr2=0x7ff7840cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff781d9b700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840cf7f0 0x7ff7840cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff781d9b700 1 -- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004020 msgr2=0x7ff784149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff781d9b700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004020 0x7ff784149e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff781d9b700 1 -- 172.21.15.89:0/586784968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78414a3d0 con 0x7ff78415e8a0 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.168+0000 7ff78259c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004020 0x7ff784149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.169+0000 7ff781d9b700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff78415e8a0 0x7ff7840cf2b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff778000c00 tx=0x7ff7780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:29.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.169+0000 7ff772ffd700 1 -- 172.21.15.89:0/586784968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7780045c0 con 0x7ff78415e8a0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.169+0000 7ff772ffd700 1 -- 172.21.15.89:0/586784968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff778020070 con 0x7ff78415e8a0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.169+0000 7ff772ffd700 1 -- 172.21.15.89:0/586784968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77801c440 con 0x7ff78415e8a0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.169+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7840ce5c0 con 0x7ff78415e8a0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.169+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78415ad80 con 0x7ff78415e8a0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.170+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78406f3b0 con 0x7ff78415e8a0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.170+0000 7ff772ffd700 1 -- 172.21.15.89:0/586784968 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff77801c5c0 con 0x7ff78415e8a0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.170+0000 7ff772ffd700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff76406af00 0x7ff76406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.171+0000 7ff772ffd700 1 -- 172.21.15.89:0/586784968 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff778089f50 con 0x7ff78415e8a0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.171+0000 7ff78259c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff76406af00 0x7ff76406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.171+0000 7ff78259c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff76406af00 0x7ff76406d3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff774000c00 tx=0x7ff774009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:29.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.175+0000 7ff772ffd700 1 -- 172.21.15.89:0/586784968 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7780551b0 con 0x7ff78415e8a0 2023-11-24T06:19:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.382+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff784067f60 con 0x7ff78415e8a0 2023-11-24T06:19:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.382+0000 7ff772ffd700 1 -- 172.21.15.89:0/586784968 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff7780587d0 con 0x7ff78415e8a0 2023-11-24T06:19:29.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.384+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff76406af00 msgr2=0x7ff76406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.384+0000 7ff788f8c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff76406af00 0x7ff76406d3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff774000c00 tx=0x7ff774009040 comp rx=0 tx=0).stop 2023-11-24T06:19:29.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff78415e8a0 msgr2=0x7ff7840cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff78415e8a0 0x7ff7840cf2b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff778000c00 tx=0x7ff7780047c0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 shutdown_connections 2023-11-24T06:19:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7840cf7f0 0x7ff7840cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff76406af00 0x7ff76406d3c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c004020 0x7ff784149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 --2- 172.21.15.89:0/586784968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff78415e8a0 0x7ff7840cf2b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 >> 172.21.15.89:0/586784968 conn(0x7ff784072990 msgr2=0x7ff7840b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 shutdown_connections 2023-11-24T06:19:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.385+0000 7ff788f8c700 1 -- 172.21.15.89:0/586784968 wait complete. 2023-11-24T06:19:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2023-11-24T06:19:29.398 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-11-24T06:19:29.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:29.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-11-24T06:19:29.444 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:19:29.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:29.444 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-24T06:19:29.504 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T06:19:29.505 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:29.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.623+0000 7fa07dfdf700 1 Processor -- start 2023-11-24T06:19:29.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.623+0000 7fa07dfdf700 1 -- start start 2023-11-24T06:19:29.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.624+0000 7fa07dfdf700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa07815ddd0 0x7fa0780b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.624+0000 7fa07dfdf700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780b1d40 con 0x7fa0780d2240 2023-11-24T06:19:29.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.624+0000 7fa07dfdf700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780b28c0 con 0x7fa07815e1b0 2023-11-24T06:19:29.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.624+0000 7fa07dfdf700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0780b1f90 con 0x7fa07815ddd0 2023-11-24T06:19:29.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.624+0000 7fa077fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa0780d2240 0x7fa07806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39912/0 (socket says 172.21.15.89:39912) 2023-11-24T06:19:29.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.624+0000 7fa077fff700 1 -- 172.21.15.89:0/1937218521 learned_addr learned my addr 172.21.15.89:0/1937218521 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:29.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0777fe700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa07815ddd0 0x7fa0780b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169163342 0 0) 0x7fa0780b1d40 con 0x7fa0780d2240 2023-11-24T06:19:29.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0600035b0 con 0x7fa0780d2240 2023-11-24T06:19:29.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3793023791 0 0) 0x7fa0600035b0 con 0x7fa0780d2240 2023-11-24T06:19:29.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa07815ddd0 msgr2=0x7fa0780b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:29.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa07815ddd0 0x7fa0780b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 >> v1:172.21.15.123:6789/0 conn(0x7fa07815e1b0 legacy=0x7fa0780d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0780b2270 con 0x7fa0780d2240 2023-11-24T06:19:29.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fa0700019f0 con 0x7fa0780d2240 2023-11-24T06:19:29.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 >> v1:172.21.15.89:6789/0 conn(0x7fa0780d2240 legacy=0x7fa07806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa07815ddd0 0x7fa060003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060003f70 0x7fa060004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa060004d90 0x7fa060005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0780b1f90 con 0x7fa060004d90 2023-11-24T06:19:29.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06000bbd0 con 0x7fa060003f70 2023-11-24T06:19:29.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06000bd90 con 0x7fa07815ddd0 2023-11-24T06:19:29.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa077fff700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa060004d90 0x7fa060005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa0777fe700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa07815ddd0 0x7fa060003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa076ffd700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060003f70 0x7fa060004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa077fff700 1 -- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa07815ddd0 msgr2=0x7fa060003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:29.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa077fff700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa07815ddd0 0x7fa060003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa077fff700 1 -- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060003f70 msgr2=0x7fa060004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:29.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa077fff700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060003f70 0x7fa060004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa077fff700 1 -- 172.21.15.89:0/1937218521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0600035b0 con 0x7fa060004d90 2023-11-24T06:19:29.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.625+0000 7fa077fff700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa060004d90 0x7fa060005210 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa07000a830 tx=0x7fa060005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3cc6050ee92c61 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:29.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070012600 con 0x7fa060004d90 2023-11-24T06:19:29.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa070012780 con 0x7fa060004d90 2023-11-24T06:19:29.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa0767fc700 1 -- 172.21.15.89:0/1937218521 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070012b30 con 0x7fa060004d90 2023-11-24T06:19:29.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa07dfdf700 1 -- 172.21.15.89:0/1937218521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa060004d90 msgr2=0x7fa060005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa07dfdf700 1 --2- 172.21.15.89:0/1937218521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa060004d90 0x7fa060005210 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa07000a830 tx=0x7fa060005880 comp rx=0 tx=0).stop 2023-11-24T06:19:29.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa077fff700 1 -- 172.21.15.89:0/1937218521 reap_dead start 2023-11-24T06:19:29.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa07dfdf700 1 -- 172.21.15.89:0/1937218521 shutdown_connections 2023-11-24T06:19:29.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa07dfdf700 1 -- 172.21.15.89:0/1937218521 >> 172.21.15.89:0/1937218521 conn(0x7fa0780731a0 msgr2=0x7fa07809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:29.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa07dfdf700 1 -- 172.21.15.89:0/1937218521 shutdown_connections 2023-11-24T06:19:29.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.626+0000 7fa07dfdf700 1 -- 172.21.15.89:0/1937218521 wait complete. 2023-11-24T06:19:29.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.628+0000 7fa07dfdf700 1 Processor -- start 2023-11-24T06:19:29.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.628+0000 7fa07dfdf700 1 -- start start 2023-11-24T06:19:29.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa07dfdf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060004d90 0x7fa0780b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa07dfdf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0781f6dd0 0x7fa0781607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa07dfdf700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0781f7250 0x7fa0781478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa07dfdf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa078160050 con 0x7fa0781f6dd0 2023-11-24T06:19:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa07dfdf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0780b7430 con 0x7fa060004d90 2023-11-24T06:19:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa07dfdf700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0780b7710 con 0x7fa0781f7250 2023-11-24T06:19:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0781f6dd0 0x7fa0781607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0781f6dd0 0x7fa0781607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36418/0 (socket says 172.21.15.89:36418) 2023-11-24T06:19:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 -- 172.21.15.89:0/2456541408 learned_addr learned my addr 172.21.15.89:0/2456541408 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa0777fe700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060004d90 0x7fa0780b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa077fff700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0781f7250 0x7fa0781478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 -- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0781f7250 msgr2=0x7fa0781478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0781f7250 0x7fa0781478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 -- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060004d90 msgr2=0x7fa0780b48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060004d90 0x7fa0780b48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 -- 172.21.15.89:0/2456541408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0780b2ed0 con 0x7fa0781f6dd0 2023-11-24T06:19:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa0777fe700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060004d90 0x7fa0780b48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa076ffd700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0781f6dd0 0x7fa0781607c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa06800f860 tx=0x7fa06800fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.629+0000 7fa05ffff700 1 -- 172.21.15.89:0/2456541408 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06800fe60 con 0x7fa0781f6dd0 2023-11-24T06:19:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.630+0000 7fa05ffff700 1 -- 172.21.15.89:0/2456541408 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0680052e0 con 0x7fa0781f6dd0 2023-11-24T06:19:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.630+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0781602a0 con 0x7fa0781f6dd0 2023-11-24T06:19:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.630+0000 7fa05ffff700 1 -- 172.21.15.89:0/2456541408 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068003680 con 0x7fa0781f6dd0 2023-11-24T06:19:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.630+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa078160ee0 con 0x7fa0781f6dd0 2023-11-24T06:19:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.631+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa078067f60 con 0x7fa0781f6dd0 2023-11-24T06:19:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.631+0000 7fa05ffff700 1 -- 172.21.15.89:0/2456541408 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa068003800 con 0x7fa0781f6dd0 2023-11-24T06:19:29.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.631+0000 7fa05ffff700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa06406aed0 0x7fa06406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:29.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.631+0000 7fa05ffff700 1 -- 172.21.15.89:0/2456541408 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa068089510 con 0x7fa0781f6dd0 2023-11-24T06:19:29.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.631+0000 7fa0777fe700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa06406aed0 0x7fa06406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:29.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.632+0000 7fa0777fe700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa06406aed0 0x7fa06406d390 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009c60 tx=0x7fa06c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:29.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.636+0000 7fa05ffff700 1 -- 172.21.15.89:0/2456541408 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa068054650 con 0x7fa0781f6dd0 2023-11-24T06:19:29.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.872+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa078160490 con 0x7fa0781f6dd0 2023-11-24T06:19:29.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.872+0000 7fa05ffff700 1 -- 172.21.15.89:0/2456541408 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fa068087e60 con 0x7fa0781f6dd0 2023-11-24T06:19:29.874 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:29.874 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:18:49.391199+0000","modified":"2023-11-24T06:18:52.422301+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:19:29.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa06406aed0 msgr2=0x7fa06406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa06406aed0 0x7fa06406d390 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009c60 tx=0x7fa06c008040 comp rx=0 tx=0).stop 2023-11-24T06:19:29.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0781f6dd0 msgr2=0x7fa0781607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:29.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0781f6dd0 0x7fa0781607c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa06800f860 tx=0x7fa06800fc20 comp rx=0 tx=0).stop 2023-11-24T06:19:29.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 shutdown_connections 2023-11-24T06:19:29.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0781f7250 0x7fa0781478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa06406aed0 0x7fa06406d390 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa060004d90 0x7fa0780b48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 --2- 172.21.15.89:0/2456541408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0781f6dd0 0x7fa0781607c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 >> 172.21.15.89:0/2456541408 conn(0x7fa0780731a0 msgr2=0x7fa07806fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 shutdown_connections 2023-11-24T06:19:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:29.874+0000 7fa07dfdf700 1 -- 172.21.15.89:0/2456541408 wait complete. 2023-11-24T06:19:29.879 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-11-24T06:19:29.889 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:30.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.006+0000 7f5174ed2700 1 Processor -- start 2023-11-24T06:19:30.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.007+0000 7f5174ed2700 1 -- start start 2023-11-24T06:19:30.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.007+0000 7f5174ed2700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158670 0x7f51700622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.007+0000 7f5174ed2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f51700c9fc0 0x7f51700ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f5174ed2700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51700cfff0 con 0x7f5170158a50 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f5174ed2700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51700cfd10 con 0x7f51700c9fc0 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f5174ed2700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5170161f50 con 0x7f5170158670 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516dd9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5170158a50 0x7f51700c98b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39930/0 (socket says 172.21.15.89:39930) 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516dd9b700 1 -- 172.21.15.89:0/2619227053 learned_addr learned my addr 172.21.15.89:0/2619227053 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516ed9d700 1 --2- 172.21.15.89:0/2619227053 >> v2:172.21.15.123:3300/0 conn(0x7f51700c9fc0 0x7f51700ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516e59c700 1 --2- 172.21.15.89:0/2619227053 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158670 0x7f51700622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516d59a700 1 -- 172.21.15.89:0/2619227053 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276941466 0 0) 0x7f51700cfff0 con 0x7f5170158a50 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516d59a700 1 -- 172.21.15.89:0/2619227053 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f515c0035b0 con 0x7f5170158a50 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516ed9d700 1 -- 172.21.15.89:0/2619227053 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158670 msgr2=0x7f51700622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516ed9d700 1 --2- 172.21.15.89:0/2619227053 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158670 0x7f51700622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516d59a700 1 -- 172.21.15.89:0/2619227053 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1201512065 0 0) 0x7f515c0035b0 con 0x7f5170158a50 2023-11-24T06:19:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516ed9d700 1 -- 172.21.15.89:0/2619227053 >> v1:172.21.15.89:6789/0 conn(0x7f5170158a50 legacy=0x7f51700c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516ed9d700 1 -- 172.21.15.89:0/2619227053 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5170162230 con 0x7f51700c9fc0 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516ed9d700 1 --2- 172.21.15.89:0/2619227053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51700c9fc0 0x7f51700ce1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5168009d90 tx=0x7f516800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5070c27c741f4699 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.008+0000 7f516d59a700 1 -- 172.21.15.89:0/2619227053 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516800e070 con 0x7f51700c9fc0 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f516d59a700 1 -- 172.21.15.89:0/2619227053 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5168003a20 con 0x7f51700c9fc0 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f516d59a700 1 -- 172.21.15.89:0/2619227053 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160001690 con 0x7f51700c9fc0 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f5174ed2700 1 -- 172.21.15.89:0/2619227053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51700c9fc0 msgr2=0x7f51700ce1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2619227053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51700c9fc0 0x7f51700ce1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5168009d90 tx=0x7f516800b330 comp rx=0 tx=0).stop 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f5174ed2700 1 -- 172.21.15.89:0/2619227053 shutdown_connections 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2619227053 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158670 0x7f51700622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2619227053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51700c9fc0 0x7f51700ce1a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f5174ed2700 1 -- 172.21.15.89:0/2619227053 >> 172.21.15.89:0/2619227053 conn(0x7f51700b57b0 msgr2=0x7f51700b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:30.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f5174ed2700 1 -- 172.21.15.89:0/2619227053 shutdown_connections 2023-11-24T06:19:30.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.009+0000 7f5174ed2700 1 -- 172.21.15.89:0/2619227053 wait complete. 2023-11-24T06:19:30.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f5174ed2700 1 Processor -- start 2023-11-24T06:19:30.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f5174ed2700 1 -- start start 2023-11-24T06:19:30.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f5174ed2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51700c9fc0 0x7f51700adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f5174ed2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5170158670 0x7f51701f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f5174ed2700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158a50 0x7f51701f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f5174ed2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51700af090 con 0x7f51700c9fc0 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f5174ed2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51700ae480 con 0x7f5170158670 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f5174ed2700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51700ae760 con 0x7f5170158a50 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.011+0000 7f516e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51700c9fc0 0x7f51700adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51700c9fc0 0x7f51700adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36432/0 (socket says 172.21.15.89:36432) 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516e59c700 1 -- 172.21.15.89:0/2184412655 learned_addr learned my addr 172.21.15.89:0/2184412655 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516dd9b700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5170158670 0x7f51701f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516ed9d700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158a50 0x7f51701f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516e59c700 1 -- 172.21.15.89:0/2184412655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158a50 msgr2=0x7f51701f74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516e59c700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158a50 0x7f51701f74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516e59c700 1 -- 172.21.15.89:0/2184412655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5170158670 msgr2=0x7f51701f6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516e59c700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5170158670 0x7f51701f6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516e59c700 1 -- 172.21.15.89:0/2184412655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5168009a70 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f516e59c700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51700c9fc0 0x7f51700adb60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f515800e9b0 tx=0x7f515800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f5166ffd700 1 -- 172.21.15.89:0/2184412655 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5158009710 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f5166ffd700 1 -- 172.21.15.89:0/2184412655 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515800c850 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51700aea40 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f5166ffd700 1 -- 172.21.15.89:0/2184412655 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515800cbc0 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.012+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51700ae0a0 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.013+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5170067f60 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.013+0000 7f5166ffd700 1 -- 172.21.15.89:0/2184412655 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f515800cd20 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.014+0000 7f5166ffd700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f515006aed0 0x7f515006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.014+0000 7f5166ffd700 1 -- 172.21.15.89:0/2184412655 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5158089730 con 0x7f51700c9fc0 2023-11-24T06:19:30.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.014+0000 7f516dd9b700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f515006aed0 0x7f515006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.014+0000 7f516dd9b700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f515006aed0 0x7f515006d390 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5160001eb0 tx=0x7f5160001c40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.018+0000 7f5166ffd700 1 -- 172.21.15.89:0/2184412655 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51580547e0 con 0x7f51700c9fc0 2023-11-24T06:19:30.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.214+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5170074a20 con 0x7f51700c9fc0 2023-11-24T06:19:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.215+0000 7f5166ffd700 1 -- 172.21.15.89:0/2184412655 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16689 (secure 0 0 0) 0x7f5158057e00 con 0x7f51700c9fc0 2023-11-24T06:19:30.218 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:30.218 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:19:30.200678+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1},{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f515006aed0 msgr2=0x7f515006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f515006aed0 0x7f515006d390 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5160001eb0 tx=0x7f5160001c40 comp rx=0 tx=0).stop 2023-11-24T06:19:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51700c9fc0 msgr2=0x7f51700adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51700c9fc0 0x7f51700adb60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f515800e9b0 tx=0x7f515800ed70 comp rx=0 tx=0).stop 2023-11-24T06:19:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 shutdown_connections 2023-11-24T06:19:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5170158a50 0x7f51701f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51700c9fc0 0x7f51700adb60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f515006aed0 0x7f515006d390 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 --2- 172.21.15.89:0/2184412655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5170158670 0x7f51701f6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 >> 172.21.15.89:0/2184412655 conn(0x7f51700b57b0 msgr2=0x7f51700b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:30.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 shutdown_connections 2023-11-24T06:19:30.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.218+0000 7f5174ed2700 1 -- 172.21.15.89:0/2184412655 wait complete. 2023-11-24T06:19:30.233 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:18:47.048442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:18:48.361659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:30.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:30.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.355+0000 7faaf95b0700 1 Processor -- start 2023-11-24T06:19:30.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.355+0000 7faaf95b0700 1 -- start start 2023-11-24T06:19:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf95b0700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faaf40d0890 0x7faaf40d3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf95b0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40d0c70 0x7faaf40b7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf95b0700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf40cf570 con 0x7faaf40cefb0 2023-11-24T06:19:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf95b0700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf40cf850 con 0x7faaf40d0890 2023-11-24T06:19:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf95b0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40d2b30 con 0x7faaf40d0c70 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf2ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faaf40d0890 0x7faaf40d3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf27fc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faaf40cefb0 0x7faaf40b3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39944/0 (socket says 172.21.15.89:39944) 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf27fc700 1 -- 172.21.15.89:0/550625649 learned_addr learned my addr 172.21.15.89:0/550625649 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf37fe700 1 --2- 172.21.15.89:0/550625649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40d0c70 0x7faaf40b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf2ffd700 1 -- 172.21.15.89:0/550625649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40d0c70 msgr2=0x7faaf40b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf2ffd700 1 --2- 172.21.15.89:0/550625649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40d0c70 0x7faaf40b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf2ffd700 1 -- 172.21.15.89:0/550625649 >> v1:172.21.15.89:6789/0 conn(0x7faaf40cefb0 legacy=0x7faaf40b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf1ffb700 1 -- 172.21.15.89:0/550625649 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402086471 0 0) 0x7faaf40cf570 con 0x7faaf40cefb0 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf2ffd700 1 -- 172.21.15.89:0/550625649 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf40d2d40 con 0x7faaf40d0890 2023-11-24T06:19:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf37fe700 1 --2- 172.21.15.89:0/550625649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40d0c70 0x7faaf40b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.356+0000 7faaf2ffd700 1 --2- 172.21.15.89:0/550625649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0890 0x7faaf40d3820 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7faae8009840 tx=0x7faae8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a449b13f33032eeb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf1ffb700 1 -- 172.21.15.89:0/550625649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8011830 con 0x7faaf40d0890 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf1ffb700 1 -- 172.21.15.89:0/550625649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae8011990 con 0x7faaf40d0890 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf1ffb700 1 -- 172.21.15.89:0/550625649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8011d20 con 0x7faaf40d0890 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf95b0700 1 -- 172.21.15.89:0/550625649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0890 msgr2=0x7faaf40d3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf95b0700 1 --2- 172.21.15.89:0/550625649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0890 0x7faaf40d3820 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7faae8009840 tx=0x7faae8009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf95b0700 1 -- 172.21.15.89:0/550625649 shutdown_connections 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf95b0700 1 --2- 172.21.15.89:0/550625649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40d0c70 0x7faaf40b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf95b0700 1 --2- 172.21.15.89:0/550625649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0890 0x7faaf40d3820 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf95b0700 1 -- 172.21.15.89:0/550625649 >> 172.21.15.89:0/550625649 conn(0x7faaf406b410 msgr2=0x7faaf406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf95b0700 1 -- 172.21.15.89:0/550625649 shutdown_connections 2023-11-24T06:19:30.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.357+0000 7faaf95b0700 1 -- 172.21.15.89:0/550625649 wait complete. 2023-11-24T06:19:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.358+0000 7faaf95b0700 1 Processor -- start 2023-11-24T06:19:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf95b0700 1 -- start start 2023-11-24T06:19:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf95b0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40cefb0 0x7faaf41533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf95b0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faaf40d0890 0x7faaf41f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf95b0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0c70 0x7faaf40aa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf95b0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40cc0d0 con 0x7faaf40d0c70 2023-11-24T06:19:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf95b0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf41538e0 con 0x7faaf40d0890 2023-11-24T06:19:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf95b0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40cc320 con 0x7faaf40cefb0 2023-11-24T06:19:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0c70 0x7faaf40aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0c70 0x7faaf40aa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36448/0 (socket says 172.21.15.89:36448) 2023-11-24T06:19:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf37fe700 1 -- 172.21.15.89:0/832928039 learned_addr learned my addr 172.21.15.89:0/832928039 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf2ffd700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40cefb0 0x7faaf41533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.359+0000 7faaf27fc700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faaf40d0890 0x7faaf41f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faaf37fe700 1 -- 172.21.15.89:0/832928039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40cefb0 msgr2=0x7faaf41533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faaf37fe700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40cefb0 0x7faaf41533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faaf37fe700 1 -- 172.21.15.89:0/832928039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faaf40d0890 msgr2=0x7faaf41f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faaf37fe700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faaf40d0890 0x7faaf41f4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faaf37fe700 1 -- 172.21.15.89:0/832928039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae80094a0 con 0x7faaf40d0c70 2023-11-24T06:19:30.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faaf37fe700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0c70 0x7faaf40aa830 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7faaec007df0 tx=0x7faaec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faadb7fe700 1 -- 172.21.15.89:0/832928039 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec00dd20 con 0x7faaf40d0c70 2023-11-24T06:19:30.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faadb7fe700 1 -- 172.21.15.89:0/832928039 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaec00f040 con 0x7faaf40d0c70 2023-11-24T06:19:30.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faadb7fe700 1 -- 172.21.15.89:0/832928039 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec003c60 con 0x7faaf40d0c70 2023-11-24T06:19:30.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf40cc600 con 0x7faaf40d0c70 2023-11-24T06:19:30.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.360+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf40ccbc0 con 0x7faaf40d0c70 2023-11-24T06:19:30.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.361+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf4068440 con 0x7faaf40d0c70 2023-11-24T06:19:30.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.361+0000 7faadb7fe700 1 -- 172.21.15.89:0/832928039 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faaec004030 con 0x7faaf40d0c70 2023-11-24T06:19:30.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.362+0000 7faadb7fe700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faae006aed0 0x7faae006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.362+0000 7faaf2ffd700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faae006aed0 0x7faae006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.362+0000 7faadb7fe700 1 -- 172.21.15.89:0/832928039 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7faaec089600 con 0x7faaf40d0c70 2023-11-24T06:19:30.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.362+0000 7faaf2ffd700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faae006aed0 0x7faae006d390 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faae80090b0 tx=0x7faae8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.366+0000 7faadb7fe700 1 -- 172.21.15.89:0/832928039 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faaec054710 con 0x7faaf40d0c70 2023-11-24T06:19:30.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.555+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faaf4068fb0 con 0x7faaf40d0c70 2023-11-24T06:19:30.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.556+0000 7faadb7fe700 1 -- 172.21.15.89:0/832928039 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7faaec057d30 con 0x7faaf40d0c70 2023-11-24T06:19:30.559 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:30.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faae006aed0 msgr2=0x7faae006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faae006aed0 0x7faae006d390 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faae80090b0 tx=0x7faae8008030 comp rx=0 tx=0).stop 2023-11-24T06:19:30.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0c70 msgr2=0x7faaf40aa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0c70 0x7faaf40aa830 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7faaec007df0 tx=0x7faaec00d3b0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 shutdown_connections 2023-11-24T06:19:30.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf40d0c70 0x7faaf40aa830 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faaf40cefb0 0x7faaf41533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7faae006aed0 0x7faae006d390 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 --2- 172.21.15.89:0/832928039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faaf40d0890 0x7faaf41f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 >> 172.21.15.89:0/832928039 conn(0x7faaf406b410 msgr2=0x7faaf40b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:30.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 shutdown_connections 2023-11-24T06:19:30.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.558+0000 7faaf95b0700 1 -- 172.21.15.89:0/832928039 wait complete. 2023-11-24T06:19:30.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:19:30.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.693+0000 7f9f5d660700 1 Processor -- start 2023-11-24T06:19:30.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.693+0000 7f9f5d660700 1 -- start start 2023-11-24T06:19:30.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f5d660700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9f5815e320 0x7f9f580d2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f5d660700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f5806b700 0x7f9f58167d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f5d660700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580b6fb0 con 0x7f9f580b93c0 2023-11-24T06:19:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f5d660700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f580d4da0 con 0x7f9f5815e320 2023-11-24T06:19:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f5d660700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580d2940 con 0x7f9f5806b700 2023-11-24T06:19:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f577fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f5806b700 0x7f9f58167d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f56ffd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9f5815e320 0x7f9f580d2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f577fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f5806b700 0x7f9f58167d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:45330/0 (socket says 172.21.15.89:45330) 2023-11-24T06:19:30.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.694+0000 7f9f577fe700 1 -- 172.21.15.89:0/1603586601 learned_addr learned my addr 172.21.15.89:0/1603586601 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:30.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f577fe700 1 -- 172.21.15.89:0/1603586601 >> v2:172.21.15.123:3300/0 conn(0x7f9f5815e320 msgr2=0x7f9f580d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:30.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f577fe700 1 --2- 172.21.15.89:0/1603586601 >> v2:172.21.15.123:3300/0 conn(0x7f9f5815e320 0x7f9f580d2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f577fe700 1 -- 172.21.15.89:0/1603586601 >> v1:172.21.15.123:6789/0 conn(0x7f9f580b93c0 legacy=0x7f9f5806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f577fe700 1 -- 172.21.15.89:0/1603586601 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f580d2c10 con 0x7f9f5806b700 2023-11-24T06:19:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f56ffd700 1 --2- 172.21.15.89:0/1603586601 >> v2:172.21.15.123:3300/0 conn(0x7f9f5815e320 0x7f9f580d2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f577fe700 1 --2- 172.21.15.89:0/1603586601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f5806b700 0x7f9f58167d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009e50 tx=0x7f9f5000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=318897ef373bee31 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f55ffb700 1 -- 172.21.15.89:0/1603586601 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5000e070 con 0x7f9f5806b700 2023-11-24T06:19:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f55ffb700 1 -- 172.21.15.89:0/1603586601 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5000b910 con 0x7f9f5806b700 2023-11-24T06:19:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.695+0000 7f9f55ffb700 1 -- 172.21.15.89:0/1603586601 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5000bcc0 con 0x7f9f5806b700 2023-11-24T06:19:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.696+0000 7f9f5d660700 1 -- 172.21.15.89:0/1603586601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f5806b700 msgr2=0x7f9f58167d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.696+0000 7f9f5d660700 1 --2- 172.21.15.89:0/1603586601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f5806b700 0x7f9f58167d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f50009e50 tx=0x7f9f5000b330 comp rx=0 tx=0).stop 2023-11-24T06:19:30.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.696+0000 7f9f5d660700 1 -- 172.21.15.89:0/1603586601 shutdown_connections 2023-11-24T06:19:30.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.696+0000 7f9f5d660700 1 --2- 172.21.15.89:0/1603586601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f5806b700 0x7f9f58167d70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.696+0000 7f9f5d660700 1 --2- 172.21.15.89:0/1603586601 >> v2:172.21.15.123:3300/0 conn(0x7f9f5815e320 0x7f9f580d2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.696+0000 7f9f5d660700 1 -- 172.21.15.89:0/1603586601 >> 172.21.15.89:0/1603586601 conn(0x7f9f580d3e50 msgr2=0x7f9f580d4260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:30.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.696+0000 7f9f5d660700 1 -- 172.21.15.89:0/1603586601 shutdown_connections 2023-11-24T06:19:30.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.696+0000 7f9f5d660700 1 -- 172.21.15.89:0/1603586601 wait complete. 2023-11-24T06:19:30.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.697+0000 7f9f5d660700 1 Processor -- start 2023-11-24T06:19:30.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.697+0000 7f9f5d660700 1 -- start start 2023-11-24T06:19:30.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f5d660700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f5806b700 0x7f9f580c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f5d660700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f580b93c0 0x7f9f58150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f5d660700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f5815e320 0x7f9f580caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f5d660700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580c9220 con 0x7f9f5815e320 2023-11-24T06:19:30.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f5d660700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580c9500 con 0x7f9f5806b700 2023-11-24T06:19:30.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f5d660700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580c97e0 con 0x7f9f580b93c0 2023-11-24T06:19:30.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f5815e320 0x7f9f580caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f5815e320 0x7f9f580caf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36460/0 (socket says 172.21.15.89:36460) 2023-11-24T06:19:30.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 -- 172.21.15.89:0/2109179767 learned_addr learned my addr 172.21.15.89:0/2109179767 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:30.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f56ffd700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f5806b700 0x7f9f580c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f567fc700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f580b93c0 0x7f9f58150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 -- 172.21.15.89:0/2109179767 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f580b93c0 msgr2=0x7f9f58150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f580b93c0 0x7f9f58150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 -- 172.21.15.89:0/2109179767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f5806b700 msgr2=0x7f9f580c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f5806b700 0x7f9f580c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 -- 172.21.15.89:0/2109179767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0094a0 con 0x7f9f5815e320 2023-11-24T06:19:30.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.698+0000 7f9f577fe700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f5815e320 0x7f9f580caf20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9f50012000 tx=0x7f9f50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.699+0000 7f9f3f7fe700 1 -- 172.21.15.89:0/2109179767 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50004800 con 0x7f9f5815e320 2023-11-24T06:19:30.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.699+0000 7f9f3f7fe700 1 -- 172.21.15.89:0/2109179767 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f50004960 con 0x7f9f5815e320 2023-11-24T06:19:30.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.699+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f50009b30 con 0x7f9f5815e320 2023-11-24T06:19:30.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.699+0000 7f9f3f7fe700 1 -- 172.21.15.89:0/2109179767 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5001c680 con 0x7f9f5815e320 2023-11-24T06:19:30.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.699+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f580c9da0 con 0x7f9f5815e320 2023-11-24T06:19:30.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.700+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f580ca110 con 0x7f9f5815e320 2023-11-24T06:19:30.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.700+0000 7f9f3f7fe700 1 -- 172.21.15.89:0/2109179767 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9f5001c7e0 con 0x7f9f5815e320 2023-11-24T06:19:30.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.700+0000 7f9f3f7fe700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9f4406aed0 0x7f9f4406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:30.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.701+0000 7f9f56ffd700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9f4406aed0 0x7f9f4406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:30.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.701+0000 7f9f3f7fe700 1 -- 172.21.15.89:0/2109179767 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f9f5008aef0 con 0x7f9f5815e320 2023-11-24T06:19:30.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.701+0000 7f9f56ffd700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9f4406aed0 0x7f9f4406d390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009cd0 tx=0x7f9f4c009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:30.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.705+0000 7f9f3f7fe700 1 -- 172.21.15.89:0/2109179767 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f50055f30 con 0x7f9f5815e320 2023-11-24T06:19:30.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.894+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f580cfc10 con 0x7f9f5815e320 2023-11-24T06:19:30.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.894+0000 7f9f3f7fe700 1 -- 172.21.15.89:0/2109179767 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9f50059550 con 0x7f9f5815e320 2023-11-24T06:19:30.896 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:19:30.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9f4406aed0 msgr2=0x7f9f4406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9f4406aed0 0x7f9f4406d390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009cd0 tx=0x7f9f4c009210 comp rx=0 tx=0).stop 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f5815e320 msgr2=0x7f9f580caf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f5815e320 0x7f9f580caf20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9f50012000 tx=0x7f9f50004ad0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 shutdown_connections 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f5815e320 0x7f9f580caf20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9f4406aed0 0x7f9f4406d390 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f5806b700 0x7f9f580c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 --2- 172.21.15.89:0/2109179767 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f580b93c0 0x7f9f58150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 >> 172.21.15.89:0/2109179767 conn(0x7f9f580d3e50 msgr2=0x7f9f580b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.896+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 shutdown_connections 2023-11-24T06:19:30.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:30.897+0000 7f9f5d660700 1 -- 172.21.15.89:0/2109179767 wait complete. 2023-11-24T06:19:30.910 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:19:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.032+0000 7f4405686700 1 Processor -- start 2023-11-24T06:19:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.032+0000 7f4405686700 1 -- start start 2023-11-24T06:19:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f4405686700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44000d45d0 0x7f44000b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f4405686700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44000d39d0 0x7f44000b7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f4405686700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000d2930 con 0x7f440015e140 2023-11-24T06:19:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f4405686700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f440015e520 con 0x7f44000d45d0 2023-11-24T06:19:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f4405686700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000d2c10 con 0x7f44000d39d0 2023-11-24T06:19:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fe7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44000d45d0 0x7f44000b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fe7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44000d45d0 0x7f44000b70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36466/0 (socket says 172.21.15.89:36466) 2023-11-24T06:19:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fe7fc700 1 -- 172.21.15.89:0/3435382113 learned_addr learned my addr 172.21.15.89:0/3435382113 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43ff7fe700 1 --2- 172.21.15.89:0/3435382113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44000d39d0 0x7f44000b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fe7fc700 1 -- 172.21.15.89:0/3435382113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44000d39d0 msgr2=0x7f44000b7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fe7fc700 1 --2- 172.21.15.89:0/3435382113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44000d39d0 0x7f44000b7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fe7fc700 1 -- 172.21.15.89:0/3435382113 >> v1:172.21.15.123:6789/0 conn(0x7f440015e140 legacy=0x7f44000d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fe7fc700 1 -- 172.21.15.89:0/3435382113 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000d2ef0 con 0x7f44000d45d0 2023-11-24T06:19:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fe7fc700 1 --2- 172.21.15.89:0/3435382113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44000b70d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f440015e660 tx=0x7f43f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3591688a6487d1c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fdffb700 1 -- 172.21.15.89:0/3435382113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0004230 con 0x7f44000d45d0 2023-11-24T06:19:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.033+0000 7f43fdffb700 1 -- 172.21.15.89:0/3435382113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f0004390 con 0x7f44000d45d0 2023-11-24T06:19:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f43fdffb700 1 -- 172.21.15.89:0/3435382113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f00036a0 con 0x7f44000d45d0 2023-11-24T06:19:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f4405686700 1 -- 172.21.15.89:0/3435382113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 msgr2=0x7f44000b70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f4405686700 1 --2- 172.21.15.89:0/3435382113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44000b70d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f440015e660 tx=0x7f43f0009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f4405686700 1 -- 172.21.15.89:0/3435382113 shutdown_connections 2023-11-24T06:19:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f4405686700 1 --2- 172.21.15.89:0/3435382113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44000d39d0 0x7f44000b7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f4405686700 1 --2- 172.21.15.89:0/3435382113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44000b70d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f4405686700 1 -- 172.21.15.89:0/3435382113 >> 172.21.15.89:0/3435382113 conn(0x7f440006b410 msgr2=0x7f440006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f4405686700 1 -- 172.21.15.89:0/3435382113 shutdown_connections 2023-11-24T06:19:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.034+0000 7f4405686700 1 -- 172.21.15.89:0/3435382113 wait complete. 2023-11-24T06:19:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.035+0000 7f4405686700 1 Processor -- start 2023-11-24T06:19:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.035+0000 7f4405686700 1 -- start start 2023-11-24T06:19:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f4405686700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44000d39d0 0x7f4400157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f4405686700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44001f7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f4405686700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440015e140 0x7f44000c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f4405686700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44000b2530 con 0x7f44000d45d0 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f4405686700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44001f72a0 con 0x7f44000d39d0 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f4405686700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000cba10 con 0x7f440015e140 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f43fe7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44001f7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f43fe7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44001f7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36472/0 (socket says 172.21.15.89:36472) 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f43fe7fc700 1 -- 172.21.15.89:0/2243606100 learned_addr learned my addr 172.21.15.89:0/2243606100 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f43feffd700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44000d39d0 0x7f4400157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f43fe7fc700 1 -- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440015e140 msgr2=0x7f44000c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.036+0000 7f43ff7fe700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440015e140 0x7f44000c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43fe7fc700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440015e140 0x7f44000c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43fe7fc700 1 -- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44000d39d0 msgr2=0x7f4400157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43fe7fc700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44000d39d0 0x7f4400157ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43fe7fc700 1 -- 172.21.15.89:0/2243606100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0009550 con 0x7f44000d45d0 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43feffd700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44000d39d0 0x7f4400157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43fe7fc700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44001f7a60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009130 tx=0x7f43f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43e77fe700 1 -- 172.21.15.89:0/2243606100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0003e20 con 0x7f44000d45d0 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43e77fe700 1 -- 172.21.15.89:0/2243606100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f000b670 con 0x7f44000d45d0 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44000cbcf0 con 0x7f44000d45d0 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f43e77fe700 1 -- 172.21.15.89:0/2243606100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f000ba00 con 0x7f44000d45d0 2023-11-24T06:19:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.037+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44001f7540 con 0x7f44000d45d0 2023-11-24T06:19:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.038+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4400068fb0 con 0x7f44000d45d0 2023-11-24T06:19:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.038+0000 7f43e77fe700 1 -- 172.21.15.89:0/2243606100 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f43f0021070 con 0x7f44000d45d0 2023-11-24T06:19:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.039+0000 7f43e77fe700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f43ec06aed0 0x7f43ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.039+0000 7f43e77fe700 1 -- 172.21.15.89:0/2243606100 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f43f008afb0 con 0x7f44000d45d0 2023-11-24T06:19:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.039+0000 7f43feffd700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f43ec06aed0 0x7f43ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.039+0000 7f43feffd700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f43ec06aed0 0x7f43ec06d390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f43f4000f60 tx=0x7f43f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.043+0000 7f43e77fe700 1 -- 172.21.15.89:0/2243606100 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43f00bde90 con 0x7f44000d45d0 2023-11-24T06:19:31.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.236+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f440015d730 con 0x7f44000d45d0 2023-11-24T06:19:31.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.236+0000 7f43e77fe700 1 -- 172.21.15.89:0/2243606100 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43f00290a0 con 0x7f44000d45d0 2023-11-24T06:19:31.238 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f43ec06aed0 msgr2=0x7f43ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f43ec06aed0 0x7f43ec06d390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f43f4000f60 tx=0x7f43f4009040 comp rx=0 tx=0).stop 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 msgr2=0x7f44001f7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44001f7a60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009130 tx=0x7f43f0004750 comp rx=0 tx=0).stop 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 shutdown_connections 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440015e140 0x7f44000c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f43ec06aed0 0x7f43ec06d390 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44000d39d0 0x7f4400157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 --2- 172.21.15.89:0/2243606100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44000d45d0 0x7f44001f7a60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 >> 172.21.15.89:0/2243606100 conn(0x7f440006b410 msgr2=0x7f44000d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:31.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 shutdown_connections 2023-11-24T06:19:31.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.238+0000 7f4405686700 1 -- 172.21.15.89:0/2243606100 wait complete. 2023-11-24T06:19:31.253 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.372+0000 7fc4c1106700 1 Processor -- start 2023-11-24T06:19:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.372+0000 7fc4c1106700 1 -- start start 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4c1106700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4c1106700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4bc0b3370 0x7fc4bc0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4c1106700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0b2030 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4c1106700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b2f50 con 0x7fc4bc0b3370 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4c1106700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b22d0 con 0x7fc4bc15bdd0 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4ba59c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4bc0b3370 0x7fc4bc0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4ba59c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4bc0b3370 0x7fc4bc0b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36478/0 (socket says 172.21.15.89:36478) 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4ba59c700 1 -- 172.21.15.89:0/837882051 learned_addr learned my addr 172.21.15.89:0/837882051 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4bad9d700 1 --2- 172.21.15.89:0/837882051 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.373+0000 7fc4ba59c700 1 -- 172.21.15.89:0/837882051 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 msgr2=0x7fc4bc1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4ba59c700 1 --2- 172.21.15.89:0/837882051 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc1604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4ba59c700 1 -- 172.21.15.89:0/837882051 >> v1:172.21.15.123:6789/0 conn(0x7fc4bc0d1c70 legacy=0x7fc4bc0ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4ba59c700 1 -- 172.21.15.89:0/837882051 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0b25b0 con 0x7fc4bc0b3370 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4bad9d700 1 --2- 172.21.15.89:0/837882051 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4ba59c700 1 --2- 172.21.15.89:0/837882051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0b3370 0x7fc4bc0b1af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0098f0 tx=0x7fc4ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722c61e236ecaeaf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4b9d9b700 1 -- 172.21.15.89:0/837882051 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac00b8a0 con 0x7fc4bc0b3370 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4b9d9b700 1 -- 172.21.15.89:0/837882051 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac00ba00 con 0x7fc4bc0b3370 2023-11-24T06:19:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4b9d9b700 1 -- 172.21.15.89:0/837882051 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac00bd90 con 0x7fc4bc0b3370 2023-11-24T06:19:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4c1106700 1 -- 172.21.15.89:0/837882051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0b3370 msgr2=0x7fc4bc0b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4c1106700 1 --2- 172.21.15.89:0/837882051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0b3370 0x7fc4bc0b1af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0098f0 tx=0x7fc4ac009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4c1106700 1 -- 172.21.15.89:0/837882051 shutdown_connections 2023-11-24T06:19:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4c1106700 1 --2- 172.21.15.89:0/837882051 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4c1106700 1 --2- 172.21.15.89:0/837882051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0b3370 0x7fc4bc0b1af0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4c1106700 1 -- 172.21.15.89:0/837882051 >> 172.21.15.89:0/837882051 conn(0x7fc4bc0731a0 msgr2=0x7fc4bc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4c1106700 1 -- 172.21.15.89:0/837882051 shutdown_connections 2023-11-24T06:19:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.374+0000 7fc4c1106700 1 -- 172.21.15.89:0/837882051 wait complete. 2023-11-24T06:19:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.376+0000 7fc4c1106700 1 Processor -- start 2023-11-24T06:19:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.376+0000 7fc4c1106700 1 -- start start 2023-11-24T06:19:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4c1106700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4bc0b3370 0x7fc4bc0b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4c1106700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0d1c70 0x7fc4bc14d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4c1106700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4c1106700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0cbc10 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4c1106700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b4e10 con 0x7fc4bc0b3370 2023-11-24T06:19:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4c1106700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0cb090 con 0x7fc4bc15bdd0 2023-11-24T06:19:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0d1c70 0x7fc4bc14d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0d1c70 0x7fc4bc14d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36486/0 (socket says 172.21.15.89:36486) 2023-11-24T06:19:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 -- 172.21.15.89:0/3904239106 learned_addr learned my addr 172.21.15.89:0/3904239106 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4bad9d700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4bc0b3370 0x7fc4bc0b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4bb59e700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 -- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 msgr2=0x7fc4bc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 -- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4bc0b3370 msgr2=0x7fc4bc0b48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4bc0b3370 0x7fc4bc0b48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 -- 172.21.15.89:0/3904239106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac009550 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4bad9d700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4bc0b3370 0x7fc4bc0b48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4ba59c700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0d1c70 0x7fc4bc14d0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009c40 tx=0x7fc4ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4a37fe700 1 -- 172.21.15.89:0/3904239106 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac004530 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4a37fe700 1 -- 172.21.15.89:0/3904239106 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac01f070 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4a37fe700 1 -- 172.21.15.89:0/3904239106 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac0038e0 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc0cb370 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.377+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc0cb930 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.378+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc0c8f50 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.379+0000 7fc4a37fe700 1 -- 172.21.15.89:0/3904239106 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc4ac021030 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.379+0000 7fc4a37fe700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4a806aed0 0x7fc4a806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.379+0000 7fc4a37fe700 1 -- 172.21.15.89:0/3904239106 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fc4ac08ab00 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.379+0000 7fc4bad9d700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4a806aed0 0x7fc4a806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.380+0000 7fc4bad9d700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4a806aed0 0x7fc4a806d390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0008910 tx=0x7fc4b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.384+0000 7fc4a37fe700 1 -- 172.21.15.89:0/3904239106 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4ac055b50 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.617+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4bc067f60 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.618+0000 7fc4a37fe700 1 -- 172.21.15.89:0/3904239106 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fc4ac015090 con 0x7fc4bc0d1c70 2023-11-24T06:19:31.620 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:31.620 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:18:49.391199+0000","modified":"2023-11-24T06:18:52.422301+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:19:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4a806aed0 msgr2=0x7fc4a806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4a806aed0 0x7fc4a806d390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0008910 tx=0x7fc4b0008040 comp rx=0 tx=0).stop 2023-11-24T06:19:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0d1c70 msgr2=0x7fc4bc14d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0d1c70 0x7fc4bc14d0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009c40 tx=0x7fc4ac004750 comp rx=0 tx=0).stop 2023-11-24T06:19:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 shutdown_connections 2023-11-24T06:19:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4bc15bdd0 0x7fc4bc0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc4a806aed0 0x7fc4a806d390 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4bc0b3370 0x7fc4bc0b48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 --2- 172.21.15.89:0/3904239106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0d1c70 0x7fc4bc14d0c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 >> 172.21.15.89:0/3904239106 conn(0x7fc4bc0731a0 msgr2=0x7fc4bc0b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 shutdown_connections 2023-11-24T06:19:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.620+0000 7fc4c1106700 1 -- 172.21.15.89:0/3904239106 wait complete. 2023-11-24T06:19:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-11-24T06:19:31.634 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:31.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.752+0000 7f844ed81700 1 Processor -- start 2023-11-24T06:19:31.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.752+0000 7f844ed81700 1 -- start start 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844ed81700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f844815ae20 0x7f84480b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844ed81700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f844815b200 0x7f84480b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844ed81700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480cf780 0x7f84480b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844ed81700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84480d0020 con 0x7f844815ae20 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844ed81700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84480b3d10 con 0x7f844815b200 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844ed81700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84480cfd40 con 0x7f84480cf780 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844cb1d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f844815ae20 0x7f84480b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844cb1d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f844815ae20 0x7f84480b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36492/0 (socket says 172.21.15.89:36492) 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844cb1d700 1 -- 172.21.15.89:0/1239293672 learned_addr learned my addr 172.21.15.89:0/1239293672 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f8447fff700 1 --2- 172.21.15.89:0/1239293672 >> v2:172.21.15.123:3300/0 conn(0x7f844815b200 0x7f84480b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844d31e700 1 --2- 172.21.15.89:0/1239293672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480cf780 0x7f84480b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844cb1d700 1 -- 172.21.15.89:0/1239293672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480cf780 msgr2=0x7f84480b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844cb1d700 1 --2- 172.21.15.89:0/1239293672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480cf780 0x7f84480b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844cb1d700 1 -- 172.21.15.89:0/1239293672 >> v2:172.21.15.123:3300/0 conn(0x7f844815b200 msgr2=0x7f84480b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844cb1d700 1 --2- 172.21.15.89:0/1239293672 >> v2:172.21.15.123:3300/0 conn(0x7f844815b200 0x7f84480b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844cb1d700 1 -- 172.21.15.89:0/1239293672 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84480affa0 con 0x7f844815ae20 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.753+0000 7f844d31e700 1 --2- 172.21.15.89:0/1239293672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480cf780 0x7f84480b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844cb1d700 1 --2- 172.21.15.89:0/1239293672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f844815ae20 0x7f84480b29a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f84480cefe0 tx=0x7f843c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25182fbf996d7534 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f84477fe700 1 -- 172.21.15.89:0/1239293672 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c00b860 con 0x7f844815ae20 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f84477fe700 1 -- 172.21.15.89:0/1239293672 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c00b9c0 con 0x7f844815ae20 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f84477fe700 1 -- 172.21.15.89:0/1239293672 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c00bd50 con 0x7f844815ae20 2023-11-24T06:19:31.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 -- 172.21.15.89:0/1239293672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f844815ae20 msgr2=0x7f84480b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 --2- 172.21.15.89:0/1239293672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f844815ae20 0x7f84480b29a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f84480cefe0 tx=0x7f843c009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:31.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 -- 172.21.15.89:0/1239293672 shutdown_connections 2023-11-24T06:19:31.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 --2- 172.21.15.89:0/1239293672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480cf780 0x7f84480b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 --2- 172.21.15.89:0/1239293672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f844815ae20 0x7f84480b29a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 --2- 172.21.15.89:0/1239293672 >> v2:172.21.15.123:3300/0 conn(0x7f844815b200 0x7f84480b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 -- 172.21.15.89:0/1239293672 >> 172.21.15.89:0/1239293672 conn(0x7f84480b57b0 msgr2=0x7f84480b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:31.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 -- 172.21.15.89:0/1239293672 shutdown_connections 2023-11-24T06:19:31.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.754+0000 7f844ed81700 1 -- 172.21.15.89:0/1239293672 wait complete. 2023-11-24T06:19:31.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.755+0000 7f844ed81700 1 Processor -- start 2023-11-24T06:19:31.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844ed81700 1 -- start start 2023-11-24T06:19:31.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844ed81700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84480cf780 0x7f8448157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844ed81700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844815ae20 0x7f84480ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844ed81700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f844815b200 0x7f84480c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844ed81700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84480ad270 con 0x7f84480cf780 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844ed81700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84480aeaf0 con 0x7f844815ae20 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844ed81700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84480ad510 con 0x7f844815b200 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844cb1d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84480cf780 0x7f8448157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.756+0000 7f844cb1d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84480cf780 0x7f8448157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36496/0 (socket says 172.21.15.89:36496) 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844cb1d700 1 -- 172.21.15.89:0/3848707098 learned_addr learned my addr 172.21.15.89:0/3848707098 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f8447fff700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844815ae20 0x7f84480ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844d31e700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f844815b200 0x7f84480c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844cb1d700 1 -- 172.21.15.89:0/3848707098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f844815b200 msgr2=0x7f84480c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844cb1d700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f844815b200 0x7f84480c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844cb1d700 1 -- 172.21.15.89:0/3848707098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844815ae20 msgr2=0x7f84480ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844cb1d700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844815ae20 0x7f84480ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844cb1d700 1 -- 172.21.15.89:0/3848707098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c009550 con 0x7f84480cf780 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844cb1d700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84480cf780 0x7f8448157720 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f843c009c40 tx=0x7f843c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f84457fa700 1 -- 172.21.15.89:0/3848707098 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c004300 con 0x7f84480cf780 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f84457fa700 1 -- 172.21.15.89:0/3848707098 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c01f070 con 0x7f84480cf780 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84480ad7f0 con 0x7f84480cf780 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f84457fa700 1 -- 172.21.15.89:0/3848707098 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0036a0 con 0x7f84480cf780 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.757+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84480addb0 con 0x7f84480cf780 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.758+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8448067f60 con 0x7f84480cf780 2023-11-24T06:19:31.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.758+0000 7f84457fa700 1 -- 172.21.15.89:0/3848707098 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f843c021030 con 0x7f84480cf780 2023-11-24T06:19:31.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.759+0000 7f84457fa700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f842c06aed0 0x7f842c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:31.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.759+0000 7f84457fa700 1 -- 172.21.15.89:0/3848707098 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f843c08a930 con 0x7f84480cf780 2023-11-24T06:19:31.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.759+0000 7f8447fff700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f842c06aed0 0x7f842c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:31.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.759+0000 7f8447fff700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f842c06aed0 0x7f842c06d390 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f844814feb0 tx=0x7f8434008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:31.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.763+0000 7f84457fa700 1 -- 172.21.15.89:0/3848707098 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f843c055a10 con 0x7f84480cf780 2023-11-24T06:19:31.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.955+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84480cde80 con 0x7f84480cf780 2023-11-24T06:19:31.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.956+0000 7f84457fa700 1 -- 172.21.15.89:0/3848707098 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16689 (secure 0 0 0) 0x7f843c015020 con 0x7f84480cf780 2023-11-24T06:19:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:31.959 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:19:31.207194+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1},{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:31.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f842c06aed0 msgr2=0x7f842c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f842c06aed0 0x7f842c06d390 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f844814feb0 tx=0x7f8434008040 comp rx=0 tx=0).stop 2023-11-24T06:19:31.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84480cf780 msgr2=0x7f8448157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:31.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84480cf780 0x7f8448157720 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f843c009c40 tx=0x7f843c004750 comp rx=0 tx=0).stop 2023-11-24T06:19:31.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 shutdown_connections 2023-11-24T06:19:31.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f844815b200 0x7f84480c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84480cf780 0x7f8448157720 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f842c06aed0 0x7f842c06d390 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 --2- 172.21.15.89:0/3848707098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844815ae20 0x7f84480ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:31.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 >> 172.21.15.89:0/3848707098 conn(0x7f84480b57b0 msgr2=0x7f84480ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:31.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 shutdown_connections 2023-11-24T06:19:31.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:31.958+0000 7f844ed81700 1 -- 172.21.15.89:0/3848707098 wait complete. 2023-11-24T06:19:31.973 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T06:19:31.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:32.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.092+0000 7f5e9d774700 1 Processor -- start 2023-11-24T06:19:32.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.092+0000 7f5e9d774700 1 -- start start 2023-11-24T06:19:32.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e9d774700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5e9815e1e0 0x7f5e980d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e9d774700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e9d774700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e980d20b0 con 0x7f5e980d0db0 2023-11-24T06:19:32.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e9d774700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e980d3150 con 0x7f5e9815e1e0 2023-11-24T06:19:32.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e9d774700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e980d1240 con 0x7f5e980d3a00 2023-11-24T06:19:32.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e96ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5e9815e1e0 0x7f5e980d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e96ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5e9815e1e0 0x7f5e980d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36504/0 (socket says 172.21.15.89:36504) 2023-11-24T06:19:32.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e96ffd700 1 -- 172.21.15.89:0/4101204844 learned_addr learned my addr 172.21.15.89:0/4101204844 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:32.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e967fc700 1 --2- 172.21.15.89:0/4101204844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e96ffd700 1 -- 172.21.15.89:0/4101204844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 msgr2=0x7f5e980d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e96ffd700 1 --2- 172.21.15.89:0/4101204844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e96ffd700 1 -- 172.21.15.89:0/4101204844 >> v1:172.21.15.123:6789/0 conn(0x7f5e980d0db0 legacy=0x7f5e9806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e96ffd700 1 -- 172.21.15.89:0/4101204844 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e980d2390 con 0x7f5e9815e1e0 2023-11-24T06:19:32.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e96ffd700 1 --2- 172.21.15.89:0/4101204844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9815e1e0 0x7f5e980d34c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0098f0 tx=0x7f5e8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ea083c97deb5b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.093+0000 7f5e95ffb700 1 -- 172.21.15.89:0/4101204844 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c00b8a0 con 0x7f5e9815e1e0 2023-11-24T06:19:32.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e95ffb700 1 -- 172.21.15.89:0/4101204844 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c00ba00 con 0x7f5e9815e1e0 2023-11-24T06:19:32.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e95ffb700 1 -- 172.21.15.89:0/4101204844 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c00bd90 con 0x7f5e9815e1e0 2023-11-24T06:19:32.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e9d774700 1 -- 172.21.15.89:0/4101204844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9815e1e0 msgr2=0x7f5e980d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e9d774700 1 --2- 172.21.15.89:0/4101204844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9815e1e0 0x7f5e980d34c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0098f0 tx=0x7f5e8c009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:32.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e9d774700 1 -- 172.21.15.89:0/4101204844 shutdown_connections 2023-11-24T06:19:32.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e9d774700 1 --2- 172.21.15.89:0/4101204844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9815e1e0 0x7f5e980d34c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e9d774700 1 --2- 172.21.15.89:0/4101204844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e9d774700 1 -- 172.21.15.89:0/4101204844 >> 172.21.15.89:0/4101204844 conn(0x7f5e980b5b50 msgr2=0x7f5e980b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:32.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e9d774700 1 -- 172.21.15.89:0/4101204844 shutdown_connections 2023-11-24T06:19:32.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.094+0000 7f5e9d774700 1 -- 172.21.15.89:0/4101204844 wait complete. 2023-11-24T06:19:32.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.096+0000 7f5e9d774700 1 Processor -- start 2023-11-24T06:19:32.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.096+0000 7f5e9d774700 1 -- start start 2023-11-24T06:19:32.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e9d774700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e980d0db0 0x7f5e9814d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e9d774700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e9d774700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9815e1e0 0x7f5e980cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e9d774700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9814e230 con 0x7f5e980d0db0 2023-11-24T06:19:32.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e9d774700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9814d6d0 con 0x7f5e9815e1e0 2023-11-24T06:19:32.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e9d774700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9814d920 con 0x7f5e980d3a00 2023-11-24T06:19:32.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e980d0db0 0x7f5e9814d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e980d0db0 0x7f5e9814d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36508/0 (socket says 172.21.15.89:36508) 2023-11-24T06:19:32.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 -- 172.21.15.89:0/487033066 learned_addr learned my addr 172.21.15.89:0/487033066 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:32.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e977fe700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9815e1e0 0x7f5e980cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e967fc700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 -- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 msgr2=0x7f5e980cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 -- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9815e1e0 msgr2=0x7f5e980cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9815e1e0 0x7f5e980cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 -- 172.21.15.89:0/487033066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c009550 con 0x7f5e980d0db0 2023-11-24T06:19:32.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e967fc700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:32.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e977fe700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9815e1e0 0x7f5e980cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:32.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e96ffd700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e980d0db0 0x7f5e9814d190 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c000c00 tx=0x7f5e8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e7f7fe700 1 -- 172.21.15.89:0/487033066 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c0036a0 con 0x7f5e980d0db0 2023-11-24T06:19:32.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e7f7fe700 1 -- 172.21.15.89:0/487033066 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c01e070 con 0x7f5e980d0db0 2023-11-24T06:19:32.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e980c8040 con 0x7f5e980d0db0 2023-11-24T06:19:32.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e7f7fe700 1 -- 172.21.15.89:0/487033066 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c003a40 con 0x7f5e980d0db0 2023-11-24T06:19:32.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.097+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9814db70 con 0x7f5e980d0db0 2023-11-24T06:19:32.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.098+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e980751e0 con 0x7f5e980d0db0 2023-11-24T06:19:32.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.099+0000 7f5e7f7fe700 1 -- 172.21.15.89:0/487033066 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5e8c003ba0 con 0x7f5e980d0db0 2023-11-24T06:19:32.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.099+0000 7f5e7f7fe700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5e8406aed0 0x7f5e8406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.099+0000 7f5e967fc700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5e8406aed0 0x7f5e8406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.099+0000 7f5e7f7fe700 1 -- 172.21.15.89:0/487033066 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f5e8c020070 con 0x7f5e980d0db0 2023-11-24T06:19:32.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.100+0000 7f5e967fc700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5e8406aed0 0x7f5e8406d390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5e98062270 tx=0x7f5e88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.104+0000 7f5e7f7fe700 1 -- 172.21.15.89:0/487033066 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e8c055be0 con 0x7f5e980d0db0 2023-11-24T06:19:32.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.340+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e9814e370 con 0x7f5e980d0db0 2023-11-24T06:19:32.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.341+0000 7f5e7f7fe700 1 -- 172.21.15.89:0/487033066 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f5e9814e370 con 0x7f5e980d0db0 2023-11-24T06:19:32.343 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:32.343 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:18:49.391199+0000","modified":"2023-11-24T06:18:52.422301+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5e8406aed0 msgr2=0x7f5e8406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5e8406aed0 0x7f5e8406d390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5e98062270 tx=0x7f5e88008040 comp rx=0 tx=0).stop 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e980d0db0 msgr2=0x7f5e9814d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e980d0db0 0x7f5e9814d190 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c000c00 tx=0x7f5e8c004750 comp rx=0 tx=0).stop 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 shutdown_connections 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5e8406aed0 0x7f5e8406d390 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e9815e1e0 0x7f5e980cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e980d0db0 0x7f5e9814d190 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 --2- 172.21.15.89:0/487033066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e980d3a00 0x7f5e980cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 >> 172.21.15.89:0/487033066 conn(0x7f5e980b5b50 msgr2=0x7f5e980b84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 shutdown_connections 2023-11-24T06:19:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.343+0000 7f5e9d774700 1 -- 172.21.15.89:0/487033066 wait complete. 2023-11-24T06:19:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-11-24T06:19:32.357 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:32.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.476+0000 7f91df0d4700 1 Processor -- start 2023-11-24T06:19:32.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.476+0000 7f91df0d4700 1 -- start start 2023-11-24T06:19:32.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.476+0000 7f91df0d4700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f91d815e080 0x7f91d80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.476+0000 7f91df0d4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d806c710 0x7f91d80afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.476+0000 7f91df0d4700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d80ae430 con 0x7f91d80af4d0 2023-11-24T06:19:32.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.476+0000 7f91df0d4700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d80af0b0 con 0x7f91d815e080 2023-11-24T06:19:32.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.476+0000 7f91df0d4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80b05a0 con 0x7f91d806c710 2023-11-24T06:19:32.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d7fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f91d80af4d0 0x7f91d806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40020/0 (socket says 172.21.15.89:40020) 2023-11-24T06:19:32.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d7fff700 1 -- 172.21.15.89:0/2786574067 learned_addr learned my addr 172.21.15.89:0/2786574067 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:32.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91dce70700 1 --2- 172.21.15.89:0/2786574067 >> v2:172.21.15.123:3300/0 conn(0x7f91d815e080 0x7f91d80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91dd671700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d806c710 0x7f91d80afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510999783 0 0) 0x7f91d80ae430 con 0x7f91d80af4d0 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c80035b0 con 0x7f91d80af4d0 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218792305 0 0) 0x7f91c80035b0 con 0x7f91d80af4d0 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d806c710 msgr2=0x7f91d80afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d806c710 0x7f91d80afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 >> v2:172.21.15.123:3300/0 conn(0x7f91d815e080 msgr2=0x7f91d80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 --2- 172.21.15.89:0/2786574067 >> v2:172.21.15.123:3300/0 conn(0x7f91d815e080 0x7f91d80b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d80ae710 con 0x7f91d80af4d0 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f91c4001670 con 0x7f91d80af4d0 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 >> v1:172.21.15.89:6789/0 conn(0x7f91d80af4d0 legacy=0x7f91d806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8003a30 0x7f91c8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8004850 0x7f91c8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d806c710 0x7f91c8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80af0b0 con 0x7f91c8003a30 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80b05a0 con 0x7f91c8004850 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c800ba10 con 0x7f91d806c710 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91dce70700 1 --2- 172.21.15.89:0/2786574067 >> v2:172.21.15.123:3300/0 conn(0x7f91d815e080 0x7f91d80b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91dce70700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8003a30 0x7f91c8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91dd671700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d806c710 0x7f91c8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.477+0000 7f91d7fff700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8004850 0x7f91c8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91dce70700 1 -- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d806c710 msgr2=0x7f91c8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91dce70700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d806c710 0x7f91c8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91dce70700 1 -- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8004850 msgr2=0x7f91c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91dce70700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8004850 0x7f91c8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91dce70700 1 -- 172.21.15.89:0/2786574067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c80035b0 con 0x7f91c8003a30 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91dce70700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8003a30 0x7f91c8003e50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009d10 tx=0x7f91c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b78e466ce0496d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc009e60 con 0x7f91c8003a30 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc00e9a0 con 0x7f91c8003a30 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91d77fe700 1 -- 172.21.15.89:0/2786574067 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc00ed10 con 0x7f91c8003a30 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91df0d4700 1 -- 172.21.15.89:0/2786574067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8003a30 msgr2=0x7f91c8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91df0d4700 1 --2- 172.21.15.89:0/2786574067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c8003a30 0x7f91c8003e50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009d10 tx=0x7f91c8005880 comp rx=0 tx=0).stop 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91dd671700 1 -- 172.21.15.89:0/2786574067 reap_dead start 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91df0d4700 1 -- 172.21.15.89:0/2786574067 shutdown_connections 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.478+0000 7f91df0d4700 1 -- 172.21.15.89:0/2786574067 >> 172.21.15.89:0/2786574067 conn(0x7f91d8071b90 msgr2=0x7f91d8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:32.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.479+0000 7f91df0d4700 1 -- 172.21.15.89:0/2786574067 shutdown_connections 2023-11-24T06:19:32.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.479+0000 7f91df0d4700 1 -- 172.21.15.89:0/2786574067 wait complete. 2023-11-24T06:19:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.480+0000 7f91df0d4700 1 Processor -- start 2023-11-24T06:19:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.480+0000 7f91df0d4700 1 -- start start 2023-11-24T06:19:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91df0d4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8003a30 0x7f91d8148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91df0d4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d81497b0 0x7f91d8157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91df0d4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91d81581a0 0x7f91d8155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91df0d4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d81f37b0 con 0x7f91d81581a0 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91df0d4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8158640 con 0x7f91c8003a30 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91df0d4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d815f4b0 con 0x7f91d81497b0 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91dd671700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91d81581a0 0x7f91d8155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91dd671700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91d81581a0 0x7f91d8155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36530/0 (socket says 172.21.15.89:36530) 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91dd671700 1 -- 172.21.15.89:0/272249593 learned_addr learned my addr 172.21.15.89:0/272249593 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91dce70700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8003a30 0x7f91d8148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.481+0000 7f91d7fff700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d81497b0 0x7f91d8157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91dd671700 1 -- 172.21.15.89:0/272249593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d81497b0 msgr2=0x7f91d8157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91dd671700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d81497b0 0x7f91d8157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91dd671700 1 -- 172.21.15.89:0/272249593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8003a30 msgr2=0x7f91d8148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91dd671700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8003a30 0x7f91d8148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91dd671700 1 -- 172.21.15.89:0/272249593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d815f790 con 0x7f91d81581a0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91dd671700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91d81581a0 0x7f91d8155fc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f91d8161a70 tx=0x7f91d0011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91d57fa700 1 -- 172.21.15.89:0/272249593 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d000e040 con 0x7f91d81581a0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91d57fa700 1 -- 172.21.15.89:0/272249593 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d000f040 con 0x7f91d81581a0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91d57fa700 1 -- 172.21.15.89:0/272249593 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d000fa00 con 0x7f91d81581a0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d815fa70 con 0x7f91d81581a0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.482+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d8156500 con 0x7f91d81581a0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.483+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d80753f0 con 0x7f91d81581a0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.483+0000 7f91d57fa700 1 -- 172.21.15.89:0/272249593 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f91d000fb60 con 0x7f91d81581a0 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.484+0000 7f91d57fa700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f91bc06aed0 0x7f91bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.484+0000 7f91d57fa700 1 -- 172.21.15.89:0/272249593 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f91d0089c10 con 0x7f91d81581a0 2023-11-24T06:19:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.484+0000 7f91dce70700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f91bc06aed0 0x7f91bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.484+0000 7f91dce70700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f91bc06aed0 0x7f91bc06d390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0090b0 tx=0x7f91cc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.488+0000 7f91d57fa700 1 -- 172.21.15.89:0/272249593 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91d0054c60 con 0x7f91d81581a0 2023-11-24T06:19:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.721+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91d8067f60 con 0x7f91d81581a0 2023-11-24T06:19:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.722+0000 7f91d57fa700 1 -- 172.21.15.89:0/272249593 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f91d0058280 con 0x7f91d81581a0 2023-11-24T06:19:32.724 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:32.724 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2470353778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2470353778},{"type":"v1","addr":"172.21.15.89:6836","nonce":2470353778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3347963772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3347963772},{"type":"v1","addr":"172.21.15.89:6837","nonce":3347963772}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/415469233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":415469233},{"type":"v1","addr":"172.21.15.123:6834","nonce":415469233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:18:49.391199+0000","modified":"2023-11-24T06:18:52.422301+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/40191848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":40191848},{"type":"v1","addr":"172.21.15.123:6835","nonce":40191848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:19:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f91bc06aed0 msgr2=0x7f91bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f91bc06aed0 0x7f91bc06d390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0090b0 tx=0x7f91cc008030 comp rx=0 tx=0).stop 2023-11-24T06:19:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91d81581a0 msgr2=0x7f91d8155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91d81581a0 0x7f91d8155fc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f91d8161a70 tx=0x7f91d0011f80 comp rx=0 tx=0).stop 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 shutdown_connections 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91d81581a0 0x7f91d8155fc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f91bc06aed0 0x7f91bc06d390 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c8003a30 0x7f91d8148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 --2- 172.21.15.89:0/272249593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d81497b0 0x7f91d8157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 >> 172.21.15.89:0/272249593 conn(0x7f91d8071b90 msgr2=0x7f91d80b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 shutdown_connections 2023-11-24T06:19:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.724+0000 7f91df0d4700 1 -- 172.21.15.89:0/272249593 wait complete. 2023-11-24T06:19:32.728 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2023-11-24T06:19:32.738 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.856+0000 7fe76e040700 1 Processor -- start 2023-11-24T06:19:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.856+0000 7fe76e040700 1 -- start start 2023-11-24T06:19:32.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe76e040700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5dd0 0x7fe7680b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe76e040700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe7680b4180 0x7fe7680af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe76e040700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680b0290 con 0x7fe7680b59f0 2023-11-24T06:19:32.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe76e040700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7680b4610 con 0x7fe7680b4180 2023-11-24T06:19:32.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe76e040700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680b4820 con 0x7fe7680b5dd0 2023-11-24T06:19:32.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7677fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe7680b59f0 0x7fe76806db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40038/0 (socket says 172.21.15.89:40038) 2023-11-24T06:19:32.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7677fe700 1 -- 172.21.15.89:0/4037477603 learned_addr learned my addr 172.21.15.89:0/4037477603 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe767fff700 1 --2- 172.21.15.89:0/4037477603 >> v2:172.21.15.123:3300/0 conn(0x7fe7680b4180 0x7fe7680af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313886722 0 0) 0x7fe7680b0290 con 0x7fe7680b59f0 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7500035b0 con 0x7fe7680b59f0 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe766ffd700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5dd0 0x7fe7680b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234382186 0 0) 0x7fe7500035b0 con 0x7fe7680b59f0 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5dd0 msgr2=0x7fe7680b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5dd0 0x7fe7680b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 >> v2:172.21.15.123:3300/0 conn(0x7fe7680b4180 msgr2=0x7fe7680af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 --2- 172.21.15.89:0/4037477603 >> v2:172.21.15.123:3300/0 conn(0x7fe7680b4180 0x7fe7680af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680b4b00 con 0x7fe7680b59f0 2023-11-24T06:19:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fe758001670 con 0x7fe7680b59f0 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 >> v1:172.21.15.89:6789/0 conn(0x7fe7680b59f0 legacy=0x7fe76806db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a00 0x7fe750003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5dd0 0x7fe750005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680b4610 con 0x7fe750003a00 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680b4820 con 0x7fe750004850 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe750014220 con 0x7fe7680b5dd0 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.857+0000 7fe7677fe700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a00 0x7fe750003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe766ffd700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe767fff700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5dd0 0x7fe750005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7677fe700 1 -- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5dd0 msgr2=0x7fe750005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7677fe700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7680b5dd0 0x7fe750005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7677fe700 1 -- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750004850 msgr2=0x7fe750004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7677fe700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750004850 0x7fe750004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7677fe700 1 -- 172.21.15.89:0/4037477603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500035b0 con 0x7fe750003a00 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7677fe700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a00 0x7fe750003e20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe75800c2f0 tx=0x7fe750005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822b0e5028c76430 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75800ce30 con 0x7fe750003a00 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758005960 con 0x7fe750003a00 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.858+0000 7fe7667fc700 1 -- 172.21.15.89:0/4037477603 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758005d10 con 0x7fe750003a00 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.859+0000 7fe76e040700 1 -- 172.21.15.89:0/4037477603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a00 msgr2=0x7fe750003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.859+0000 7fe76e040700 1 --2- 172.21.15.89:0/4037477603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe750003a00 0x7fe750003e20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe75800c2f0 tx=0x7fe750005880 comp rx=0 tx=0).stop 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.859+0000 7fe767fff700 1 -- 172.21.15.89:0/4037477603 reap_dead start 2023-11-24T06:19:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.859+0000 7fe76e040700 1 -- 172.21.15.89:0/4037477603 shutdown_connections 2023-11-24T06:19:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.859+0000 7fe76e040700 1 -- 172.21.15.89:0/4037477603 >> 172.21.15.89:0/4037477603 conn(0x7fe768072990 msgr2=0x7fe76806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.859+0000 7fe76e040700 1 -- 172.21.15.89:0/4037477603 shutdown_connections 2023-11-24T06:19:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.859+0000 7fe76e040700 1 -- 172.21.15.89:0/4037477603 wait complete. 2023-11-24T06:19:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.860+0000 7fe76e040700 1 Processor -- start 2023-11-24T06:19:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.860+0000 7fe76e040700 1 -- start start 2023-11-24T06:19:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe76e040700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a00 0x7fe7681f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe76e040700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe768150f10 0x7fe7681efce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe76e040700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe768155970 0x7fe76814d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe76e040700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681f0220 con 0x7fe768155970 2023-11-24T06:19:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe76e040700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681564f0 con 0x7fe750003a00 2023-11-24T06:19:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe76e040700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe768149e40 con 0x7fe768150f10 2023-11-24T06:19:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe768155970 0x7fe76814d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe768155970 0x7fe76814d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36550/0 (socket says 172.21.15.89:36550) 2023-11-24T06:19:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 -- 172.21.15.89:0/1628623486 learned_addr learned my addr 172.21.15.89:0/1628623486 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe7677fe700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a00 0x7fe7681f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe766ffd700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe768150f10 0x7fe7681efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 -- 172.21.15.89:0/1628623486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe768150f10 msgr2=0x7fe7681efce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe768150f10 0x7fe7681efce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 -- 172.21.15.89:0/1628623486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a00 msgr2=0x7fe7681f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a00 0x7fe7681f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 -- 172.21.15.89:0/1628623486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768151390 con 0x7fe768155970 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe767fff700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe768155970 0x7fe76814d0b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe768070690 tx=0x7fe76000f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe74ffff700 1 -- 172.21.15.89:0/1628623486 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76000e040 con 0x7fe768155970 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe74ffff700 1 -- 172.21.15.89:0/1628623486 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76000f040 con 0x7fe768155970 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe768155e20 con 0x7fe768155970 2023-11-24T06:19:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe74ffff700 1 -- 172.21.15.89:0/1628623486 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760015430 con 0x7fe768155970 2023-11-24T06:19:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.861+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe768156740 con 0x7fe768155970 2023-11-24T06:19:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.862+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe768157070 con 0x7fe768155970 2023-11-24T06:19:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.863+0000 7fe74ffff700 1 -- 172.21.15.89:0/1628623486 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe760015590 con 0x7fe768155970 2023-11-24T06:19:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.863+0000 7fe74ffff700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.863+0000 7fe74ffff700 1 -- 172.21.15.89:0/1628623486 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fe760089bb0 con 0x7fe768155970 2023-11-24T06:19:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.863+0000 7fe7677fe700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.864+0000 7fe7677fe700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe758008fd0 tx=0x7fe758007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:32.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:32.867+0000 7fe74ffff700 1 -- 172.21.15.89:0/1628623486 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe760054c00 con 0x7fe768155970 2023-11-24T06:19:33.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.060+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7680ad970 con 0x7fe768155970 2023-11-24T06:19:33.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.061+0000 7fe74ffff700 1 -- 172.21.15.89:0/1628623486 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16689 (secure 0 0 0) 0x7fe760058220 con 0x7fe768155970 2023-11-24T06:19:33.063 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:33.063 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:19:31.207194+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:18:47.048442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:18:48.361659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1},{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 msgr2=0x7fe75406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe758008fd0 tx=0x7fe758007000 comp rx=0 tx=0).stop 2023-11-24T06:19:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe768155970 msgr2=0x7fe76814d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe768155970 0x7fe76814d0b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe768070690 tx=0x7fe76000f490 comp rx=0 tx=0).stop 2023-11-24T06:19:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 shutdown_connections 2023-11-24T06:19:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe768155970 0x7fe76814d0b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe75406aed0 0x7fe75406d390 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe750003a00 0x7fe7681f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 --2- 172.21.15.89:0/1628623486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe768150f10 0x7fe7681efce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 >> 172.21.15.89:0/1628623486 conn(0x7fe768072990 msgr2=0x7fe768161c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 shutdown_connections 2023-11-24T06:19:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.063+0000 7fe76e040700 1 -- 172.21.15.89:0/1628623486 wait complete. 2023-11-24T06:19:33.077 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T06:19:33.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.196+0000 7ffbec112700 1 Processor -- start 2023-11-24T06:19:33.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.196+0000 7ffbec112700 1 -- start start 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbec112700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe40b59f0 0x7ffbe406db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbec112700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffbe406e0b0 0x7ffbe40b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbec112700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40b0290 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbec112700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe40b4820 con 0x7ffbe406e0b0 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbec112700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40b4b00 con 0x7ffbe40b59f0 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbea6af700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ffbe40b5dd0 0x7ffbe40af190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40060/0 (socket says 172.21.15.89:40060) 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbea6af700 1 -- 172.21.15.89:0/1199837435 learned_addr learned my addr 172.21.15.89:0/1199837435 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbe9eae700 1 --2- 172.21.15.89:0/1199837435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe40b59f0 0x7ffbe406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbe96ad700 1 --2- 172.21.15.89:0/1199837435 >> v2:172.21.15.123:3300/0 conn(0x7ffbe406e0b0 0x7ffbe40b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.197+0000 7ffbe8eac700 1 -- 172.21.15.89:0/1199837435 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811991381 0 0) 0x7ffbe40b0290 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe8eac700 1 -- 172.21.15.89:0/1199837435 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd00035b0 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe96ad700 1 -- 172.21.15.89:0/1199837435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe40b59f0 msgr2=0x7ffbe406db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe96ad700 1 --2- 172.21.15.89:0/1199837435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe40b59f0 0x7ffbe406db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe96ad700 1 -- 172.21.15.89:0/1199837435 >> v1:172.21.15.89:6789/0 conn(0x7ffbe40b5dd0 legacy=0x7ffbe40af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe96ad700 1 -- 172.21.15.89:0/1199837435 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe40ae9b0 con 0x7ffbe406e0b0 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe96ad700 1 --2- 172.21.15.89:0/1199837435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe406e0b0 0x7ffbe40b3c40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009840 tx=0x7ffbe0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=791230a6ce2949b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe8eac700 1 -- 172.21.15.89:0/1199837435 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0011830 con 0x7ffbe406e0b0 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe8eac700 1 -- 172.21.15.89:0/1199837435 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe0011990 con 0x7ffbe406e0b0 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbe8eac700 1 -- 172.21.15.89:0/1199837435 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0011d00 con 0x7ffbe406e0b0 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.198+0000 7ffbec112700 1 -- 172.21.15.89:0/1199837435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe406e0b0 msgr2=0x7ffbe40b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.199+0000 7ffbec112700 1 --2- 172.21.15.89:0/1199837435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe406e0b0 0x7ffbe40b3c40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009840 tx=0x7ffbe0009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.199+0000 7ffbec112700 1 -- 172.21.15.89:0/1199837435 shutdown_connections 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.199+0000 7ffbec112700 1 --2- 172.21.15.89:0/1199837435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe40b59f0 0x7ffbe406db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.199+0000 7ffbec112700 1 --2- 172.21.15.89:0/1199837435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe406e0b0 0x7ffbe40b3c40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.199+0000 7ffbec112700 1 -- 172.21.15.89:0/1199837435 >> 172.21.15.89:0/1199837435 conn(0x7ffbe4072990 msgr2=0x7ffbe406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.199+0000 7ffbec112700 1 -- 172.21.15.89:0/1199837435 shutdown_connections 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.199+0000 7ffbec112700 1 -- 172.21.15.89:0/1199837435 wait complete. 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.200+0000 7ffbec112700 1 Processor -- start 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.200+0000 7ffbec112700 1 -- start start 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.200+0000 7ffbec112700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe406e0b0 0x7ffbe41f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.200+0000 7ffbec112700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe40b59f0 0x7ffbe414e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbec112700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbe40b5dd0 0x7ffbe4152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbec112700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe41f83f0 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbec112700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe41f4e10 con 0x7ffbe40b59f0 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbec112700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe41f7870 con 0x7ffbe406e0b0 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbe40b5dd0 0x7ffbe4152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbe40b5dd0 0x7ffbe4152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36564/0 (socket says 172.21.15.89:36564) 2023-11-24T06:19:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 -- 172.21.15.89:0/3270872473 learned_addr learned my addr 172.21.15.89:0/3270872473 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbe96ad700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe40b59f0 0x7ffbe414e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbe9eae700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe406e0b0 0x7ffbe41f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 -- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe406e0b0 msgr2=0x7ffbe41f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe406e0b0 0x7ffbe41f48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 -- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe40b59f0 msgr2=0x7ffbe414e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe40b59f0 0x7ffbe414e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 -- 172.21.15.89:0/3270872473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe00094a0 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbe9eae700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe406e0b0 0x7ffbe41f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbea6af700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbe40b5dd0 0x7ffbe4152960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00caa0 tx=0x7ffbdc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbda7fc700 1 -- 172.21.15.89:0/3270872473 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00aa80 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbda7fc700 1 -- 172.21.15.89:0/3270872473 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc00abe0 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe41f7b50 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbda7fc700 1 -- 172.21.15.89:0/3270872473 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc012400 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.201+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe41f8110 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.202+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe406c5e0 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.203+0000 7ffbda7fc700 1 -- 172.21.15.89:0/3270872473 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffbdc01b3b0 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.203+0000 7ffbda7fc700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffbc806aed0 0x7ffbc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.203+0000 7ffbda7fc700 1 -- 172.21.15.89:0/3270872473 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5804+0+0 (secure 0 0 0) 0x7ffbdc089390 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.203+0000 7ffbe9eae700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffbc806aed0 0x7ffbc806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.203+0000 7ffbe9eae700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffbc806aed0 0x7ffbc806d390 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40cd920 tx=0x7ffbd4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.207+0000 7ffbda7fc700 1 -- 172.21.15.89:0/3270872473 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffbdc0543e0 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.440+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffbe4067f60 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.443 INFO:tasks.ceph.mon.a.smithi089.stderr:2023-11-24T06:19:33.441+0000 7f7eccc4c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T06:19:33.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.442+0000 7ffbda7fc700 1 -- 172.21.15.89:0/3270872473 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffbdc057a00 con 0x7ffbe40b5dd0 2023-11-24T06:19:33.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffbc806aed0 msgr2=0x7ffbc806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffbc806aed0 0x7ffbc806d390 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40cd920 tx=0x7ffbd4008040 comp rx=0 tx=0).stop 2023-11-24T06:19:33.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbe40b5dd0 msgr2=0x7ffbe4152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbe40b5dd0 0x7ffbe4152960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00caa0 tx=0x7ffbdc00ce60 comp rx=0 tx=0).stop 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 shutdown_connections 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbe40b5dd0 0x7ffbe4152960 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbe406e0b0 0x7ffbe41f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffbc806aed0 0x7ffbc806d390 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 --2- 172.21.15.89:0/3270872473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbe40b59f0 0x7ffbe414e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 >> 172.21.15.89:0/3270872473 conn(0x7ffbe4072990 msgr2=0x7ffbe40b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 shutdown_connections 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.445+0000 7ffbec112700 1 -- 172.21.15.89:0/3270872473 wait complete. 2023-11-24T06:19:33.448 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T06:19:33.459 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T06:19:33.487 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2023-11-24T06:19:33.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.579+0000 7f333b503700 1 Processor -- start 2023-11-24T06:19:33.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.579+0000 7f333b503700 1 -- start start 2023-11-24T06:19:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.580+0000 7f333b503700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3334151c30 0x7f33340ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.580+0000 7f333b503700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33340b55a0 con 0x7f333406fab0 2023-11-24T06:19:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.580+0000 7f333b503700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334062900 con 0x7f33340cf0b0 2023-11-24T06:19:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.580+0000 7f333b503700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33340b52c0 con 0x7f3334151c30 2023-11-24T06:19:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.580+0000 7f3338a9e700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f333406fab0 0x7f33340ce9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40074/0 (socket says 172.21.15.89:40074) 2023-11-24T06:19:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.580+0000 7f3338a9e700 1 -- 172.21.15.89:0/3577051200 learned_addr learned my addr 172.21.15.89:0/3577051200 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f333929f700 1 --2- 172.21.15.89:0/3577051200 >> v2:172.21.15.123:3300/0 conn(0x7f3334151c30 0x7f33340ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f332bfff700 1 -- 172.21.15.89:0/3577051200 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230738136 0 0) 0x7f33340b55a0 con 0x7f333406fab0 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f332bfff700 1 -- 172.21.15.89:0/3577051200 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33200035b0 con 0x7f333406fab0 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f332bfff700 1 -- 172.21.15.89:0/3577051200 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539721515 0 0) 0x7f3334062900 con 0x7f33340cf0b0 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f332bfff700 1 -- 172.21.15.89:0/3577051200 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33340b55a0 con 0x7f33340cf0b0 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f333929f700 1 -- 172.21.15.89:0/3577051200 >> v1:172.21.15.123:6789/0 conn(0x7f33340cf0b0 legacy=0x7f333415e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f333929f700 1 -- 172.21.15.89:0/3577051200 >> v1:172.21.15.89:6789/0 conn(0x7f333406fab0 legacy=0x7f33340ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f333929f700 1 -- 172.21.15.89:0/3577051200 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334062340 con 0x7f3334151c30 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f333929f700 1 --2- 172.21.15.89:0/3577051200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3334151c30 0x7f33340ccfb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3324009840 tx=0x7f3324009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f04528fcd804127 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f332bfff700 1 -- 172.21.15.89:0/3577051200 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324011830 con 0x7f3334151c30 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f332bfff700 1 -- 172.21.15.89:0/3577051200 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3324011990 con 0x7f3334151c30 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.581+0000 7f332bfff700 1 -- 172.21.15.89:0/3577051200 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324011d20 con 0x7f3334151c30 2023-11-24T06:19:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.582+0000 7f333b503700 1 -- 172.21.15.89:0/3577051200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3334151c30 msgr2=0x7f33340ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.582+0000 7f333b503700 1 --2- 172.21.15.89:0/3577051200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3334151c30 0x7f33340ccfb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3324009840 tx=0x7f3324009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:33.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.582+0000 7f333b503700 1 -- 172.21.15.89:0/3577051200 shutdown_connections 2023-11-24T06:19:33.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.582+0000 7f333b503700 1 --2- 172.21.15.89:0/3577051200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3334151c30 0x7f33340ccfb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.582+0000 7f333b503700 1 -- 172.21.15.89:0/3577051200 >> 172.21.15.89:0/3577051200 conn(0x7f33340b2050 msgr2=0x7f33340b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:33.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.582+0000 7f333b503700 1 -- 172.21.15.89:0/3577051200 shutdown_connections 2023-11-24T06:19:33.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.582+0000 7f333b503700 1 -- 172.21.15.89:0/3577051200 wait complete. 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.583+0000 7f333b503700 1 Processor -- start 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.583+0000 7f333b503700 1 -- start start 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f333b503700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333406fab0 0x7f33341547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f333b503700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf0b0 0x7f33340b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f333b503700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334151c30 0x7f33340b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f333b503700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3334154ce0 con 0x7f3334151c30 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f333b503700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3334155860 con 0x7f333406fab0 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f333b503700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3334155b40 con 0x7f33340cf0b0 2023-11-24T06:19:33.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334151c30 0x7f33340b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334151c30 0x7f33340b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36578/0 (socket says 172.21.15.89:36578) 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 -- 172.21.15.89:0/2566013150 learned_addr learned my addr 172.21.15.89:0/2566013150 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f333929f700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333406fab0 0x7f33341547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3338a9e700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf0b0 0x7f33340b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 -- 172.21.15.89:0/2566013150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf0b0 msgr2=0x7f33340b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf0b0 0x7f33340b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 -- 172.21.15.89:0/2566013150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333406fab0 msgr2=0x7f33341547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333406fab0 0x7f33341547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 -- 172.21.15.89:0/2566013150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33240094a0 con 0x7f3334151c30 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3339aa0700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334151c30 0x7f33340b16f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f332c00cbc0 tx=0x7f332c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:33.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3329ffb700 1 -- 172.21.15.89:0/2566013150 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c003c00 con 0x7f3334151c30 2023-11-24T06:19:33.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3329ffb700 1 -- 172.21.15.89:0/2566013150 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332c003d60 con 0x7f3334151c30 2023-11-24T06:19:33.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3334155e20 con 0x7f3334151c30 2023-11-24T06:19:33.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.584+0000 7f3329ffb700 1 -- 172.21.15.89:0/2566013150 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c00aaa0 con 0x7f3334151c30 2023-11-24T06:19:33.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.585+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33341563e0 con 0x7f3334151c30 2023-11-24T06:19:33.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.585+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3334156a30 con 0x7f3334151c30 2023-11-24T06:19:33.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.586+0000 7f3329ffb700 1 -- 172.21.15.89:0/2566013150 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f332c00ac00 con 0x7f3334151c30 2023-11-24T06:19:33.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.586+0000 7f3329ffb700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f331806aed0 0x7f331806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:33.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.586+0000 7f3329ffb700 1 -- 172.21.15.89:0/2566013150 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f332c089630 con 0x7f3334151c30 2023-11-24T06:19:33.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.586+0000 7f333929f700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f331806aed0 0x7f331806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:33.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.587+0000 7f333929f700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f331806aed0 0x7f331806d390 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3324009fd0 tx=0x7f332401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:33.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.590+0000 7f3329ffb700 1 -- 172.21.15.89:0/2566013150 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f332c054620 con 0x7f3334151c30 2023-11-24T06:19:33.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:33.827+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f33340afd30 con 0x7f3334151c30 2023-11-24T06:19:34.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.446+0000 7f3329ffb700 1 -- 172.21.15.89:0/2566013150 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f332c05e070 con 0x7f3334151c30 2023-11-24T06:19:34.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f331806aed0 msgr2=0x7f331806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:34.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f331806aed0 0x7f331806d390 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3324009fd0 tx=0x7f332401c040 comp rx=0 tx=0).stop 2023-11-24T06:19:34.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334151c30 msgr2=0x7f33340b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:34.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334151c30 0x7f33340b16f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f332c00cbc0 tx=0x7f332c00ced0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 shutdown_connections 2023-11-24T06:19:34.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334151c30 0x7f33340b16f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f331806aed0 0x7f331806d390 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333406fab0 0x7f33341547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 --2- 172.21.15.89:0/2566013150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33340cf0b0 0x7f33340b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 >> 172.21.15.89:0/2566013150 conn(0x7f33340b2050 msgr2=0x7f333414fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:34.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 shutdown_connections 2023-11-24T06:19:34.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.449+0000 7f333b503700 1 -- 172.21.15.89:0/2566013150 wait complete. 2023-11-24T06:19:34.463 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T06:19:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.586+0000 7ff4d6363700 1 Processor -- start 2023-11-24T06:19:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.586+0000 7ff4d6363700 1 -- start start 2023-11-24T06:19:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d6363700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 0x7ff4d00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d6363700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff4d00d0420 0x7ff4d00d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d6363700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4d0157870 0x7ff4d0157d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d6363700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d00d11b0 con 0x7ff4d0157870 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d6363700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d00ce7b0 con 0x7ff4d00d0420 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d6363700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00ad2a0 con 0x7ff4d00cde90 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4d0157870 0x7ff4d0157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4d0157870 0x7ff4d0157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36586/0 (socket says 172.21.15.89:36586) 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 -- 172.21.15.89:0/1638922236 learned_addr learned my addr 172.21.15.89:0/1638922236 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4cffff700 1 --2- 172.21.15.89:0/1638922236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 0x7ff4d00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4cf7fe700 1 --2- 172.21.15.89:0/1638922236 >> v2:172.21.15.123:3300/0 conn(0x7ff4d00d0420 0x7ff4d00d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 -- 172.21.15.89:0/1638922236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 msgr2=0x7ff4d00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 --2- 172.21.15.89:0/1638922236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 0x7ff4d00ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 -- 172.21.15.89:0/1638922236 >> v2:172.21.15.123:3300/0 conn(0x7ff4d00d0420 msgr2=0x7ff4d00d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 --2- 172.21.15.89:0/1638922236 >> v2:172.21.15.123:3300/0 conn(0x7ff4d00d0420 0x7ff4d00d0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 -- 172.21.15.89:0/1638922236 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d00ad580 con 0x7ff4d0157870 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.587+0000 7ff4d4900700 1 --2- 172.21.15.89:0/1638922236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d0157870 0x7ff4d0157d00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0070ce0 tx=0x7ff4c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb20e1b1eb3984b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4ceffd700 1 -- 172.21.15.89:0/1638922236 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c800e070 con 0x7ff4d0157870 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4ceffd700 1 -- 172.21.15.89:0/1638922236 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c800b9c0 con 0x7ff4d0157870 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4ceffd700 1 -- 172.21.15.89:0/1638922236 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c800bd70 con 0x7ff4d0157870 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 -- 172.21.15.89:0/1638922236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d0157870 msgr2=0x7ff4d0157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 --2- 172.21.15.89:0/1638922236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d0157870 0x7ff4d0157d00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0070ce0 tx=0x7ff4c800b330 comp rx=0 tx=0).stop 2023-11-24T06:19:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 -- 172.21.15.89:0/1638922236 shutdown_connections 2023-11-24T06:19:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 --2- 172.21.15.89:0/1638922236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d0157870 0x7ff4d0157d00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 --2- 172.21.15.89:0/1638922236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 0x7ff4d00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 --2- 172.21.15.89:0/1638922236 >> v2:172.21.15.123:3300/0 conn(0x7ff4d00d0420 0x7ff4d00d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 -- 172.21.15.89:0/1638922236 >> 172.21.15.89:0/1638922236 conn(0x7ff4d0072710 msgr2=0x7ff4d0072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 -- 172.21.15.89:0/1638922236 shutdown_connections 2023-11-24T06:19:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.588+0000 7ff4d6363700 1 -- 172.21.15.89:0/1638922236 wait complete. 2023-11-24T06:19:34.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.589+0000 7ff4d6363700 1 Processor -- start 2023-11-24T06:19:34.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.589+0000 7ff4d6363700 1 -- start start 2023-11-24T06:19:34.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4d6363700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 0x7ff4d00b5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:34.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4d6363700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d00d0420 0x7ff4d00ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:34.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4d6363700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d0157870 0x7ff4d01f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:34.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4d6363700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00b5cc0 con 0x7ff4d00d0420 2023-11-24T06:19:34.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4d6363700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00cbf20 con 0x7ff4d0157870 2023-11-24T06:19:34.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4d6363700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00cc200 con 0x7ff4d00cde90 2023-11-24T06:19:34.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4cf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d00d0420 0x7ff4d00ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:34.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4cf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d00d0420 0x7ff4d00ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36590/0 (socket says 172.21.15.89:36590) 2023-11-24T06:19:34.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4cf7fe700 1 -- 172.21.15.89:0/4169469980 learned_addr learned my addr 172.21.15.89:0/4169469980 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:34.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4cffff700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 0x7ff4d00b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:34.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.590+0000 7ff4d4900700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d0157870 0x7ff4d01f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:34.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4cf7fe700 1 -- 172.21.15.89:0/4169469980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 msgr2=0x7ff4d00b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:34.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4cf7fe700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 0x7ff4d00b5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4cf7fe700 1 -- 172.21.15.89:0/4169469980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d0157870 msgr2=0x7ff4d01f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:34.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4cf7fe700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d0157870 0x7ff4d01f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:34.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4cf7fe700 1 -- 172.21.15.89:0/4169469980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c8009b30 con 0x7ff4d00d0420 2023-11-24T06:19:34.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4cf7fe700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d00d0420 0x7ff4d00ccce0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000d900 tx=0x7ff4c000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:34.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4ccff9700 1 -- 172.21.15.89:0/4169469980 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0011870 con 0x7ff4d00d0420 2023-11-24T06:19:34.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4ccff9700 1 -- 172.21.15.89:0/4169469980 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c00119d0 con 0x7ff4d00d0420 2023-11-24T06:19:34.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d00cc4e0 con 0x7ff4d00d0420 2023-11-24T06:19:34.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4ccff9700 1 -- 172.21.15.89:0/4169469980 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0011d80 con 0x7ff4d00d0420 2023-11-24T06:19:34.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.591+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d00cbc10 con 0x7ff4d00d0420 2023-11-24T06:19:34.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.592+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d00c6d90 con 0x7ff4d00d0420 2023-11-24T06:19:34.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.592+0000 7ff4ccff9700 1 -- 172.21.15.89:0/4169469980 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff4c000f400 con 0x7ff4d00d0420 2023-11-24T06:19:34.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.593+0000 7ff4ccff9700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff4bc06af00 0x7ff4bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:34.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.593+0000 7ff4ccff9700 1 -- 172.21.15.89:0/4169469980 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5890+0+0 (secure 0 0 0) 0x7ff4c0089570 con 0x7ff4d00d0420 2023-11-24T06:19:34.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.593+0000 7ff4cffff700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff4bc06af00 0x7ff4bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:34.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.593+0000 7ff4cffff700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff4bc06af00 0x7ff4bc06d3c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4006b20 tx=0x7ff4c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:34.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.597+0000 7ff4ccff9700 1 -- 172.21.15.89:0/4169469980 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4c0054560 con 0x7ff4d00d0420 2023-11-24T06:19:34.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:34.789+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff4d0067f60 con 0x7ff4d00d0420 2023-11-24T06:19:35.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.447+0000 7ff4ccff9700 1 -- 172.21.15.89:0/4169469980 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff4c0057b80 con 0x7ff4d00d0420 2023-11-24T06:19:35.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.450+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff4bc06af00 msgr2=0x7ff4bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:35.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.450+0000 7ff4d6363700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff4bc06af00 0x7ff4bc06d3c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4006b20 tx=0x7ff4c4008040 comp rx=0 tx=0).stop 2023-11-24T06:19:35.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.450+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d00d0420 msgr2=0x7ff4d00ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:35.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.450+0000 7ff4d6363700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d00d0420 0x7ff4d00ccce0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000d900 tx=0x7ff4c000dc10 comp rx=0 tx=0).stop 2023-11-24T06:19:35.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.451+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 shutdown_connections 2023-11-24T06:19:35.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.451+0000 7ff4d6363700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff4bc06af00 0x7ff4bc06d3c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.451+0000 7ff4d6363700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d0157870 0x7ff4d01f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.451+0000 7ff4d6363700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d00cde90 0x7ff4d00b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.451+0000 7ff4d6363700 1 --2- 172.21.15.89:0/4169469980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4d00d0420 0x7ff4d00ccce0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.451+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 >> 172.21.15.89:0/4169469980 conn(0x7ff4d0072710 msgr2=0x7ff4d01561e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:35.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.451+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 shutdown_connections 2023-11-24T06:19:35.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.451+0000 7ff4d6363700 1 -- 172.21.15.89:0/4169469980 wait complete. 2023-11-24T06:19:35.455 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2023-11-24T06:19:35.467 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T06:19:35.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.592+0000 7fdf98cc9700 1 Processor -- start 2023-11-24T06:19:35.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.592+0000 7fdf98cc9700 1 -- start start 2023-11-24T06:19:35.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf98cc9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdf9406f480 0x7fdf9406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:35.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf98cc9700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf9406e830 0x7fdf9406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:35.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf98cc9700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9406f1c0 con 0x7fdf94144600 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf98cc9700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9406aff0 con 0x7fdf9406e830 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf98cc9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9406b200 con 0x7fdf9406f480 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf92d9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf9406e830 0x7fdf9406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf92d9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf9406e830 0x7fdf9406ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36600/0 (socket says 172.21.15.89:36600) 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf92d9d700 1 -- 172.21.15.89:0/2183901849 learned_addr learned my addr 172.21.15.89:0/2183901849 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf91d9b700 1 --2- 172.21.15.89:0/2183901849 >> v2:172.21.15.123:3300/0 conn(0x7fdf9406f480 0x7fdf9406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf92d9d700 1 -- 172.21.15.89:0/2183901849 >> v2:172.21.15.123:3300/0 conn(0x7fdf9406f480 msgr2=0x7fdf9406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf92d9d700 1 --2- 172.21.15.89:0/2183901849 >> v2:172.21.15.123:3300/0 conn(0x7fdf9406f480 0x7fdf9406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf92d9d700 1 -- 172.21.15.89:0/2183901849 >> v1:172.21.15.123:6789/0 conn(0x7fdf94144600 legacy=0x7fdf940b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf92d9d700 1 -- 172.21.15.89:0/2183901849 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9406b4e0 con 0x7fdf9406e830 2023-11-24T06:19:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.593+0000 7fdf92d9d700 1 --2- 172.21.15.89:0/2183901849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406e830 0x7fdf9406ec80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdf9415b260 tx=0x7fdf8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f86d9453e29d0fe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf9159a700 1 -- 172.21.15.89:0/2183901849 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00e070 con 0x7fdf9406e830 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf9159a700 1 -- 172.21.15.89:0/2183901849 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c00b9c0 con 0x7fdf9406e830 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf9159a700 1 -- 172.21.15.89:0/2183901849 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00bd70 con 0x7fdf9406e830 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf98cc9700 1 -- 172.21.15.89:0/2183901849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406e830 msgr2=0x7fdf9406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/2183901849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406e830 0x7fdf9406ec80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdf9415b260 tx=0x7fdf8c00b330 comp rx=0 tx=0).stop 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf98cc9700 1 -- 172.21.15.89:0/2183901849 shutdown_connections 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/2183901849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406e830 0x7fdf9406ec80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/2183901849 >> v2:172.21.15.123:3300/0 conn(0x7fdf9406f480 0x7fdf9406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf98cc9700 1 -- 172.21.15.89:0/2183901849 >> 172.21.15.89:0/2183901849 conn(0x7fdf940d0b40 msgr2=0x7fdf940d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf98cc9700 1 -- 172.21.15.89:0/2183901849 shutdown_connections 2023-11-24T06:19:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.594+0000 7fdf98cc9700 1 -- 172.21.15.89:0/2183901849 wait complete. 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.596+0000 7fdf98cc9700 1 Processor -- start 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.596+0000 7fdf98cc9700 1 -- start start 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.596+0000 7fdf98cc9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf9406e830 0x7fdf94159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.596+0000 7fdf98cc9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406f480 0x7fdf9415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.596+0000 7fdf98cc9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf94144600 0x7fdf9413fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.596+0000 7fdf98cc9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9415c040 con 0x7fdf9406f480 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.596+0000 7fdf98cc9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9415c320 con 0x7fdf94144600 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.596+0000 7fdf98cc9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9415c600 con 0x7fdf9406e830 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406f480 0x7fdf9415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406f480 0x7fdf9415b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36604/0 (socket says 172.21.15.89:36604) 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 -- 172.21.15.89:0/4127235880 learned_addr learned my addr 172.21.15.89:0/4127235880 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf92d9d700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf94144600 0x7fdf9413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf9259c700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf9406e830 0x7fdf94159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 -- 172.21.15.89:0/4127235880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf9406e830 msgr2=0x7fdf94159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf9406e830 0x7fdf94159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 -- 172.21.15.89:0/4127235880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf94144600 msgr2=0x7fdf9413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf94144600 0x7fdf9413fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 -- 172.21.15.89:0/4127235880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c009b30 con 0x7fdf9406f480 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf91d9b700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406f480 0x7fdf9415b6d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800e810 tx=0x7fdf8800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf82ffd700 1 -- 172.21.15.89:0/4127235880 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8800c070 con 0x7fdf9406f480 2023-11-24T06:19:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf82ffd700 1 -- 172.21.15.89:0/4127235880 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88013610 con 0x7fdf9406f480 2023-11-24T06:19:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9415c8e0 con 0x7fdf9406f480 2023-11-24T06:19:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf82ffd700 1 -- 172.21.15.89:0/4127235880 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf880139c0 con 0x7fdf9406f480 2023-11-24T06:19:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.597+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9415a4c0 con 0x7fdf9406f480 2023-11-24T06:19:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.598+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf94067f60 con 0x7fdf9406f480 2023-11-24T06:19:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.599+0000 7fdf82ffd700 1 -- 172.21.15.89:0/4127235880 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdf88013b20 con 0x7fdf9406f480 2023-11-24T06:19:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.599+0000 7fdf82ffd700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf7406aed0 0x7fdf7406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.599+0000 7fdf9259c700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf7406aed0 0x7fdf7406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.599+0000 7fdf82ffd700 1 -- 172.21.15.89:0/4127235880 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5452+0+0 (secure 0 0 0) 0x7fdf88089320 con 0x7fdf9406f480 2023-11-24T06:19:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.600+0000 7fdf9259c700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf7406aed0 0x7fdf7406d390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdf84007fd0 tx=0x7fdf84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:35.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.604+0000 7fdf82ffd700 1 -- 172.21.15.89:0/4127235880 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf880543b0 con 0x7fdf9406f480 2023-11-24T06:19:35.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:35.801+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdf9406e0e0 con 0x7fdf9406f480 2023-11-24T06:19:36.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.452+0000 7fdf82ffd700 1 -- 172.21.15.89:0/4127235880 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdf880579d0 con 0x7fdf9406f480 2023-11-24T06:19:36.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf7406aed0 msgr2=0x7fdf7406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:36.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf7406aed0 0x7fdf7406d390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdf84007fd0 tx=0x7fdf84009040 comp rx=0 tx=0).stop 2023-11-24T06:19:36.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406f480 msgr2=0x7fdf9415b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:36.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406f480 0x7fdf9415b6d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800e810 tx=0x7fdf8800eb20 comp rx=0 tx=0).stop 2023-11-24T06:19:36.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 shutdown_connections 2023-11-24T06:19:36.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf7406aed0 0x7fdf7406d390 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf94144600 0x7fdf9413fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf9406e830 0x7fdf94159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 --2- 172.21.15.89:0/4127235880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf9406f480 0x7fdf9415b6d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 >> 172.21.15.89:0/4127235880 conn(0x7fdf940d0b40 msgr2=0x7fdf94073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:36.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 shutdown_connections 2023-11-24T06:19:36.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.456+0000 7fdf98cc9700 1 -- 172.21.15.89:0/4127235880 wait complete. 2023-11-24T06:19:36.460 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2023-11-24T06:19:36.474 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T06:19:36.475 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T06:19:36.475 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T06:19:36.475 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:19:36.475 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T06:19:36.475 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T06:19:36.476 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T06:19:36.476 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:19:36.476 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T06:19:36.476 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T06:19:36.477 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T06:19:36.477 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:19:36.477 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T06:19:36.477 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T06:19:36.477 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T06:19:36.477 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:19:36.485 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T06:19:36.485 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T06:19:36.486 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T06:19:36.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T06:19:36.487 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T06:19:36.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T06:19:36.488 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T06:19:36.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T06:19:36.527 INFO:tasks.ceph.mds.a:Started 2023-11-24T06:19:36.527 DEBUG:teuthology.parallel:result is None 2023-11-24T06:19:36.528 INFO:tasks.ceph.mds.c:Started 2023-11-24T06:19:36.528 DEBUG:teuthology.parallel:result is None 2023-11-24T06:19:36.530 INFO:tasks.ceph.mds.d:Started 2023-11-24T06:19:36.530 DEBUG:teuthology.parallel:result is None 2023-11-24T06:19:36.531 INFO:tasks.ceph.mds.b:Started 2023-11-24T06:19:36.531 DEBUG:teuthology.parallel:result is None 2023-11-24T06:19:36.531 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T06:19:36.609 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2023-11-24T06:19:36.609 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-11-24T06:19:36.619 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2023-11-24T06:19:36.619 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2023-11-24T06:19:36.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.658+0000 7f7b15cb3700 1 Processor -- start 2023-11-24T06:19:36.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.658+0000 7f7b15cb3700 1 -- start start 2023-11-24T06:19:36.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b15cb3700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7b1015e080 0x7f7b100b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b15cb3700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b100ae430 con 0x7f7b1006c710 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b15cb3700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b100ae710 con 0x7f7b100af4d0 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b15cb3700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b100af0b0 con 0x7f7b1015e080 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b0ffff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7b1006c710 0x7f7b1006e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40142/0 (socket says 172.21.15.89:40142) 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b0ffff700 1 -- 172.21.15.89:0/2450571342 learned_addr learned my addr 172.21.15.89:0/2450571342 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b0f7fe700 1 --2- 172.21.15.89:0/2450571342 >> v2:172.21.15.123:3300/0 conn(0x7f7b1015e080 0x7f7b100b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178817994 0 0) 0x7f7b100ae430 con 0x7f7b1006c710 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.659+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af80035b0 con 0x7f7b1006c710 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 38127473 0 0) 0x7f7af80035b0 con 0x7f7b1006c710 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 >> v2:172.21.15.123:3300/0 conn(0x7f7b1015e080 msgr2=0x7f7b100b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 --2- 172.21.15.89:0/2450571342 >> v2:172.21.15.123:3300/0 conn(0x7f7b1015e080 0x7f7b100b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 >> v1:172.21.15.123:6789/0 conn(0x7f7b100af4d0 legacy=0x7f7b1006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b100ae9f0 con 0x7f7b1006c710 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f7b080019f0 con 0x7f7b1006c710 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 >> v1:172.21.15.89:6789/0 conn(0x7f7b1006c710 legacy=0x7f7b1006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7af8003a00 0x7f7af8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7af8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af8005210 0x7f7af800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100af0b0 con 0x7f7af8004850 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af800c0b0 con 0x7f7af8005210 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af800c270 con 0x7f7af8003a00 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0effd700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7af8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0ffff700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af8005210 0x7f7af800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0f7fe700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7af8003a00 0x7f7af8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0effd700 1 -- 172.21.15.89:0/2450571342 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7af8003a00 msgr2=0x7f7af8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0effd700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7af8003a00 0x7f7af8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0effd700 1 -- 172.21.15.89:0/2450571342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af8005210 msgr2=0x7f7af800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0effd700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af8005210 0x7f7af800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0ffff700 1 -- 172.21.15.89:0/2450571342 reap_dead start 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0effd700 1 -- 172.21.15.89:0/2450571342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af80035b0 con 0x7f7af8004850 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0effd700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7af8004cd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7b100af950 tx=0x7f7af8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282f0b2db052302b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b000052d0 con 0x7f7af8004850 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b00005450 con 0x7f7af8004850 2023-11-24T06:19:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.660+0000 7f7b0e7fc700 1 -- 172.21.15.89:0/2450571342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b000038c0 con 0x7f7af8004850 2023-11-24T06:19:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.661+0000 7f7b15cb3700 1 -- 172.21.15.89:0/2450571342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 msgr2=0x7f7af8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.661+0000 7f7b15cb3700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7af8004cd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7b100af950 tx=0x7f7af8005880 comp rx=0 tx=0).stop 2023-11-24T06:19:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.661+0000 7f7b15cb3700 1 -- 172.21.15.89:0/2450571342 shutdown_connections 2023-11-24T06:19:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.661+0000 7f7b15cb3700 1 --2- 172.21.15.89:0/2450571342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7af8004cd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.661+0000 7f7b15cb3700 1 -- 172.21.15.89:0/2450571342 >> 172.21.15.89:0/2450571342 conn(0x7f7b10071b90 msgr2=0x7f7b10071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.661+0000 7f7b15cb3700 1 -- 172.21.15.89:0/2450571342 shutdown_connections 2023-11-24T06:19:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.661+0000 7f7b15cb3700 1 -- 172.21.15.89:0/2450571342 wait complete. 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.662+0000 7f7b15cb3700 1 Processor -- start 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.662+0000 7f7b15cb3700 1 -- start start 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b15cb3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7b10148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b15cb3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b101f6fe0 0x7f7b101f7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b15cb3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b100ce820 0x7f7b100cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b15cb3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10148ab0 con 0x7f7af8004850 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b15cb3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b101f79a0 con 0x7f7b100ce820 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b15cb3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100cf5c0 con 0x7f7b101f6fe0 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b0f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7b10148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b0f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7b10148570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36646/0 (socket says 172.21.15.89:36646) 2023-11-24T06:19:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b0f7fe700 1 -- 172.21.15.89:0/4080633464 learned_addr learned my addr 172.21.15.89:0/4080633464 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b0ffff700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b100ce820 0x7f7b100cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.663+0000 7f7b0effd700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b101f6fe0 0x7f7b101f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7b0f7fe700 1 -- 172.21.15.89:0/4080633464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b101f6fe0 msgr2=0x7f7b101f7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7b0f7fe700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b101f6fe0 0x7f7b101f7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7b0f7fe700 1 -- 172.21.15.89:0/4080633464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b100ce820 msgr2=0x7f7b100cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7b0f7fe700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b100ce820 0x7f7b100cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7b0f7fe700 1 -- 172.21.15.89:0/4080633464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b100cf8a0 con 0x7f7af8004850 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7b0f7fe700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7b10148570 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7b04011aa0 tx=0x7f7b04011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7af7fff700 1 -- 172.21.15.89:0/4080633464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b04011ee0 con 0x7f7af8004850 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7af7fff700 1 -- 172.21.15.89:0/4080633464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0400f860 con 0x7f7af8004850 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b100cfb80 con 0x7f7af8004850 2023-11-24T06:19:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7af7fff700 1 -- 172.21.15.89:0/4080633464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0400fc30 con 0x7f7af8004850 2023-11-24T06:19:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.664+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b100cf210 con 0x7f7af8004850 2023-11-24T06:19:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.665+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b100753f0 con 0x7f7af8004850 2023-11-24T06:19:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.665+0000 7f7af7fff700 1 -- 172.21.15.89:0/4080633464 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7b0400fd90 con 0x7f7af8004850 2023-11-24T06:19:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.666+0000 7f7af7fff700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7afc06aed0 0x7f7afc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.666+0000 7f7b0effd700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7afc06aed0 0x7f7afc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.666+0000 7f7af7fff700 1 -- 172.21.15.89:0/4080633464 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7b04089850 con 0x7f7af8004850 2023-11-24T06:19:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.666+0000 7f7b0effd700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7afc06aed0 0x7f7afc06d390 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7b00005040 tx=0x7f7b00009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.670+0000 7f7af7fff700 1 -- 172.21.15.89:0/4080633464 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b04054cc0 con 0x7f7af8004850 2023-11-24T06:19:36.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:36.886+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7b100d00a0 con 0x7f7af8004850 2023-11-24T06:19:37.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.461+0000 7f7af7fff700 1 -- 172.21.15.89:0/4080633464 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7b040582e0 con 0x7f7af8004850 2023-11-24T06:19:37.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7afc06aed0 msgr2=0x7f7afc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7afc06aed0 0x7f7afc06d390 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7b00005040 tx=0x7f7b00009030 comp rx=0 tx=0).stop 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 msgr2=0x7f7b10148570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7b10148570 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7b04011aa0 tx=0x7f7b04011db0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 shutdown_connections 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7afc06aed0 0x7f7afc06d390 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b100ce820 0x7f7b100cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af8004850 0x7f7b10148570 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 --2- 172.21.15.89:0/4080633464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b101f6fe0 0x7f7b101f7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 >> 172.21.15.89:0/4080633464 conn(0x7f7b10071b90 msgr2=0x7f7b1006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.464+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 shutdown_connections 2023-11-24T06:19:37.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.465+0000 7f7b15cb3700 1 -- 172.21.15.89:0/4080633464 wait complete. 2023-11-24T06:19:37.468 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2023-11-24T06:19:37.482 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T06:19:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.603+0000 7fb890717700 1 Processor -- start 2023-11-24T06:19:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.603+0000 7fb890717700 1 -- start start 2023-11-24T06:19:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb890717700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb888158a50 0x7fb8880c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb890717700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb888162a30 0x7fb8880c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb890717700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8880cfff0 con 0x7fb888158670 2023-11-24T06:19:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb890717700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8880629b0 con 0x7fb888162a30 2023-11-24T06:19:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb890717700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8880cfd10 con 0x7fb888158a50 2023-11-24T06:19:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88e4b3700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb888158670 0x7fb8880622a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40160/0 (socket says 172.21.15.89:40160) 2023-11-24T06:19:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88e4b3700 1 -- 172.21.15.89:0/2057224443 learned_addr learned my addr 172.21.15.89:0/2057224443 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88ecb4700 1 --2- 172.21.15.89:0/2057224443 >> v2:172.21.15.89:3300/0 conn(0x7fb888162a30 0x7fb8880c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88dcb2700 1 --2- 172.21.15.89:0/2057224443 >> v2:172.21.15.123:3300/0 conn(0x7fb888158a50 0x7fb8880c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88d4b1700 1 -- 172.21.15.89:0/2057224443 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740211701 0 0) 0x7fb8880cfff0 con 0x7fb888158670 2023-11-24T06:19:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88d4b1700 1 -- 172.21.15.89:0/2057224443 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8740035b0 con 0x7fb888158670 2023-11-24T06:19:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88ecb4700 1 -- 172.21.15.89:0/2057224443 >> v2:172.21.15.123:3300/0 conn(0x7fb888158a50 msgr2=0x7fb8880c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88ecb4700 1 --2- 172.21.15.89:0/2057224443 >> v2:172.21.15.123:3300/0 conn(0x7fb888158a50 0x7fb8880c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88ecb4700 1 -- 172.21.15.89:0/2057224443 >> v1:172.21.15.89:6789/0 conn(0x7fb888158670 legacy=0x7fb8880622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88ecb4700 1 -- 172.21.15.89:0/2057224443 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb888161f50 con 0x7fb888162a30 2023-11-24T06:19:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.604+0000 7fb88ecb4700 1 --2- 172.21.15.89:0/2057224443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888162a30 0x7fb8880c9df0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb8880ca730 tx=0x7fb88000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824bcddcf443a86e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb88d4b1700 1 -- 172.21.15.89:0/2057224443 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88000e070 con 0x7fb888162a30 2023-11-24T06:19:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb88d4b1700 1 -- 172.21.15.89:0/2057224443 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88000bb20 con 0x7fb888162a30 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb88d4b1700 1 -- 172.21.15.89:0/2057224443 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb880012430 con 0x7fb888162a30 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb890717700 1 -- 172.21.15.89:0/2057224443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888162a30 msgr2=0x7fb8880c9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb890717700 1 --2- 172.21.15.89:0/2057224443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888162a30 0x7fb8880c9df0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb8880ca730 tx=0x7fb88000b330 comp rx=0 tx=0).stop 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb890717700 1 -- 172.21.15.89:0/2057224443 shutdown_connections 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb890717700 1 --2- 172.21.15.89:0/2057224443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888162a30 0x7fb8880c9df0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb890717700 1 --2- 172.21.15.89:0/2057224443 >> v2:172.21.15.123:3300/0 conn(0x7fb888158a50 0x7fb8880c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb890717700 1 -- 172.21.15.89:0/2057224443 >> 172.21.15.89:0/2057224443 conn(0x7fb8880b57b0 msgr2=0x7fb8880b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb890717700 1 -- 172.21.15.89:0/2057224443 shutdown_connections 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.605+0000 7fb890717700 1 -- 172.21.15.89:0/2057224443 wait complete. 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.606+0000 7fb890717700 1 Processor -- start 2023-11-24T06:19:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.606+0000 7fb890717700 1 -- start start 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb890717700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb888158670 0x7fb888151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb890717700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888158a50 0x7fb88815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb890717700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb888162a30 0x7fb8880b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb890717700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8881f8120 con 0x7fb888158a50 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb890717700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8881f3af0 con 0x7fb888158670 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb890717700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8881f74a0 con 0x7fb888162a30 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb88dcb2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888158a50 0x7fb88815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb88dcb2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888158a50 0x7fb88815dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36664/0 (socket says 172.21.15.89:36664) 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb88dcb2700 1 -- 172.21.15.89:0/1078118950 learned_addr learned my addr 172.21.15.89:0/1078118950 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb88e4b3700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb888158670 0x7fb888151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.607+0000 7fb88ecb4700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb888162a30 0x7fb8880b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:37.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb88dcb2700 1 -- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb888162a30 msgr2=0x7fb8880b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb88dcb2700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb888162a30 0x7fb8880b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb88dcb2700 1 -- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb888158670 msgr2=0x7fb888151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb88dcb2700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb888158670 0x7fb888151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb88dcb2700 1 -- 172.21.15.89:0/1078118950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb880009a70 con 0x7fb888158a50 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb88ecb4700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb888162a30 0x7fb8880b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb88dcb2700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888158a50 0x7fb88815dc00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb87800e810 tx=0x7fb87800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb87effd700 1 -- 172.21.15.89:0/1078118950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87800c070 con 0x7fb888158a50 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb87effd700 1 -- 172.21.15.89:0/1078118950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb878013610 con 0x7fb888158a50 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8881f7780 con 0x7fb888158a50 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb87effd700 1 -- 172.21.15.89:0/1078118950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8780139c0 con 0x7fb888158a50 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.608+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8881f7d40 con 0x7fb888158a50 2023-11-24T06:19:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.609+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb888155130 con 0x7fb888158a50 2023-11-24T06:19:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.609+0000 7fb87effd700 1 -- 172.21.15.89:0/1078118950 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb878013b20 con 0x7fb888158a50 2023-11-24T06:19:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.609+0000 7fb87effd700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb86c06aed0 0x7fb86c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.610+0000 7fb87effd700 1 -- 172.21.15.89:0/1078118950 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb878088f80 con 0x7fb888158a50 2023-11-24T06:19:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.610+0000 7fb88e4b3700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb86c06aed0 0x7fb86c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.610+0000 7fb88e4b3700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb86c06aed0 0x7fb86c06d390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb884007fd0 tx=0x7fb884007dc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.614+0000 7fb87effd700 1 -- 172.21.15.89:0/1078118950 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb888155130 con 0x7fb888158a50 2023-11-24T06:19:37.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.882+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb888155130 con 0x7fb888158a50 2023-11-24T06:19:37.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.882+0000 7fb87effd700 1 -- 172.21.15.89:0/1078118950 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fb888155130 con 0x7fb888158a50 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlZPhnExAAdSzrCYJj/BqJINrd+mxnog==", 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:19:37.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlAvtyHxAAiLxnKUSDAvkxFP6LHLBkSQ==", 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:19:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlrHqfGhAA7hKW48g57uUyHOiS4gQkOQ==", 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBl4PG2JhAAg8KlX9jNiQ/Ajvrm71BM8g==", 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2023-11-24T06:19:37.888 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQASQGBlLUQyBxAApB8r9XPrC9s6zsd2nPQFoQ==", 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVQGBlFgtPBhAAvSzzgIHzad8zv8vS3fcFQg==", 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:19:37.889 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAYQGBlNW1sBRAAtOtIYUOAa63AiLIS9EYCFA==", 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAbQGBlZOIsBRAATWs4ayExofw/56n0ixjgDA==", 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAhQGBlGkIpCRAAWpMWRhHqDSEuyM+DAQLP8Q==", 2023-11-24T06:19:37.891 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAkQGBli6LfCBAAH5LBKEpbvU48rAyWnyH0Zg==", 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.892 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAnQGBl6qpjCBAASFc7C3OIr9G6lSXCNyLk8A==", 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2023-11-24T06:19:37.893 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAqQGBlxmJdCBAANGRCESck0mrD1gVuToRRaw==", 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlwIfyKhAAMZcwOQGAm+XcH+mUKcTiUg==", 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:19:37.894 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlxNQDNhAApU+JeKCeMTDo61+wi3ynbw==", 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:19:37.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBluCSzMhAA1hxy9u3cvwTZLG/Xg3kQWQ==", 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.896 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlin9NORAAkdx6VeU4rwbgNfVPwwUXFg==", 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQANQGBl+ReMMhAARm9F19thbXVu988u1kapsw==", 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.897 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-mds", 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAwQGBl2y3/BxAAq1dthLMobPUHrafrBUhC6Q==", 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-mds" 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-mgr", 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAwQGBlzDb/BxAATW53i5AVvAyk2618rjSUuw==", 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-mgr" 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-osd", 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAwQGBlw0D/BxAAvUGXWedCx5+rgS4JTNBRLg==", 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-osd" 2023-11-24T06:19:37.899 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-rbd", 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAwQGBla0j/BxAAe/BTWm1uMb/Ia0oWUSdA4A==", 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-rbd" 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAwQGBlP1D/BxAAsKrlihKxAQ2zeBkZ6E2Cmw==", 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.900 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-rgw", 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAwQGBlCVj/BxAAAHWiPbze0HKd/ovcIfYQgA==", 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-rgw" 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2023-11-24T06:19:37.901 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlj+EGEBAAFs6HBnMAVB9ZLdn7WrWpzQ==", 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlsrk+DBAAdgV15cObrTUfehK6bUY8DA==", 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:19:37.902 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2023-11-24T06:19:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:19:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:37.903 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-11-24T06:19:37.903 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:19:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb86c06aed0 msgr2=0x7fb86c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb86c06aed0 0x7fb86c06d390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb884007fd0 tx=0x7fb884007dc0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888158a50 msgr2=0x7fb88815dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888158a50 0x7fb88815dc00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb87800e810 tx=0x7fb87800eb20 comp rx=0 tx=0).stop 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 shutdown_connections 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb888162a30 0x7fb8880b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb86c06aed0 0x7fb86c06d390 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb888158670 0x7fb888151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 --2- 172.21.15.89:0/1078118950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb888158a50 0x7fb88815dc00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 >> 172.21.15.89:0/1078118950 conn(0x7fb8880b57b0 msgr2=0x7fb8880c9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 shutdown_connections 2023-11-24T06:19:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:37.885+0000 7fb890717700 1 -- 172.21.15.89:0/1078118950 wait complete. 2023-11-24T06:19:37.906 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-11-24T06:19:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.025+0000 7f64e2631700 1 Processor -- start 2023-11-24T06:19:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.025+0000 7f64e2631700 1 -- start start 2023-11-24T06:19:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.025+0000 7f64e2631700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.025+0000 7f64e2631700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc0af090 con 0x7f64dc0746a0 2023-11-24T06:19:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64e2631700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc06ec70 con 0x7f64dc06fa40 2023-11-24T06:19:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64e2631700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc06dda0 con 0x7f64dc0afc10 2023-11-24T06:19:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64e0bce700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f64dc0746a0 0x7f64dc162120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40178/0 (socket says 172.21.15.89:40178) 2023-11-24T06:19:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64e0bce700 1 -- 172.21.15.89:0/3465449633 learned_addr learned my addr 172.21.15.89:0/3465449633 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64dbfff700 1 --2- 172.21.15.89:0/3465449633 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64daffd700 1 -- 172.21.15.89:0/3465449633 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895687145 0 0) 0x7f64dc0af090 con 0x7f64dc0746a0 2023-11-24T06:19:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64daffd700 1 -- 172.21.15.89:0/3465449633 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c40035b0 con 0x7f64dc0746a0 2023-11-24T06:19:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64daffd700 1 -- 172.21.15.89:0/3465449633 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630667725 0 0) 0x7f64dc06ec70 con 0x7f64dc06fa40 2023-11-24T06:19:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64daffd700 1 -- 172.21.15.89:0/3465449633 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64dc0af090 con 0x7f64dc06fa40 2023-11-24T06:19:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64dbfff700 1 -- 172.21.15.89:0/3465449633 >> v1:172.21.15.123:6789/0 conn(0x7f64dc06fa40 legacy=0x7f64dc073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64dbfff700 1 -- 172.21.15.89:0/3465449633 >> v1:172.21.15.89:6789/0 conn(0x7f64dc0746a0 legacy=0x7f64dc162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64dbfff700 1 -- 172.21.15.89:0/3465449633 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc06e060 con 0x7f64dc0afc10 2023-11-24T06:19:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64dbfff700 1 --2- 172.21.15.89:0/3465449633 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc15e850 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009870 tx=0x7f64cc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1725635dd2a82c91 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.026+0000 7f64daffd700 1 -- 172.21.15.89:0/3465449633 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc011830 con 0x7f64dc0afc10 2023-11-24T06:19:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64daffd700 1 -- 172.21.15.89:0/3465449633 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64cc011990 con 0x7f64dc0afc10 2023-11-24T06:19:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64daffd700 1 -- 172.21.15.89:0/3465449633 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc011d00 con 0x7f64dc0afc10 2023-11-24T06:19:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64e2631700 1 -- 172.21.15.89:0/3465449633 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 msgr2=0x7f64dc15e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64e2631700 1 --2- 172.21.15.89:0/3465449633 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc15e850 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009870 tx=0x7f64cc009b80 comp rx=0 tx=0).stop 2023-11-24T06:19:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64e2631700 1 -- 172.21.15.89:0/3465449633 shutdown_connections 2023-11-24T06:19:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64e2631700 1 --2- 172.21.15.89:0/3465449633 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc15e850 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64e2631700 1 -- 172.21.15.89:0/3465449633 >> 172.21.15.89:0/3465449633 conn(0x7f64dc06aff0 msgr2=0x7f64dc06b400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:38.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64e2631700 1 -- 172.21.15.89:0/3465449633 shutdown_connections 2023-11-24T06:19:38.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.027+0000 7f64e2631700 1 -- 172.21.15.89:0/3465449633 wait complete. 2023-11-24T06:19:38.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.028+0000 7f64e2631700 1 Processor -- start 2023-11-24T06:19:38.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.028+0000 7f64e2631700 1 -- start start 2023-11-24T06:19:38.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64e2631700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc06fa40 0x7f64dc159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64e2631700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc0746a0 0x7f64dc145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64e2631700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64e2631700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc159ff0 con 0x7f64dc06fa40 2023-11-24T06:19:38.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64e2631700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc145b60 con 0x7f64dc0746a0 2023-11-24T06:19:38.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64e2631700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc145d90 con 0x7f64dc0afc10 2023-11-24T06:19:38.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc06fa40 0x7f64dc159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc06fa40 0x7f64dc159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36676/0 (socket says 172.21.15.89:36676) 2023-11-24T06:19:38.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64dbfff700 1 -- 172.21.15.89:0/342619902 learned_addr learned my addr 172.21.15.89:0/342619902 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:38.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64db7fe700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc0746a0 0x7f64dc145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.029+0000 7f64e0bce700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64dbfff700 1 -- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 msgr2=0x7f64dc146e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64dbfff700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc146e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64dbfff700 1 -- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc0746a0 msgr2=0x7f64dc145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64dbfff700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc0746a0 0x7f64dc145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64dbfff700 1 -- 172.21.15.89:0/342619902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc0094a0 con 0x7f64dc06fa40 2023-11-24T06:19:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64db7fe700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc0746a0 0x7f64dc145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64dbfff700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc06fa40 0x7f64dc159ab0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009bc0 tx=0x7f64cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64d8ff9700 1 -- 172.21.15.89:0/342619902 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0045c0 con 0x7f64dc06fa40 2023-11-24T06:19:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64d8ff9700 1 -- 172.21.15.89:0/342619902 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64cc01e070 con 0x7f64dc06fa40 2023-11-24T06:19:38.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64dc146070 con 0x7f64dc06fa40 2023-11-24T06:19:38.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64d8ff9700 1 -- 172.21.15.89:0/342619902 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc003900 con 0x7f64dc06fa40 2023-11-24T06:19:38.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.030+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64dc146630 con 0x7f64dc06fa40 2023-11-24T06:19:38.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.031+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64dc0afff0 con 0x7f64dc06fa40 2023-11-24T06:19:38.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.032+0000 7f64d8ff9700 1 -- 172.21.15.89:0/342619902 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f64cc003a60 con 0x7f64dc06fa40 2023-11-24T06:19:38.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.032+0000 7f64d8ff9700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64c806aed0 0x7f64c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.032+0000 7f64d8ff9700 1 -- 172.21.15.89:0/342619902 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f64cc08a530 con 0x7f64dc06fa40 2023-11-24T06:19:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.032+0000 7f64db7fe700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64c806aed0 0x7f64c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.033+0000 7f64db7fe700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64c806aed0 0x7f64c806d390 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f64dc14bbc0 tx=0x7f64d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.036+0000 7f64d8ff9700 1 -- 172.21.15.89:0/342619902 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64cc055aa0 con 0x7f64dc06fa40 2023-11-24T06:19:38.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.308+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f64dc0af220 con 0x7f64dc06fa40 2023-11-24T06:19:38.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.310+0000 7f64d8ff9700 1 -- 172.21.15.89:0/342619902 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f64cc0590c0 con 0x7f64dc06fa40 2023-11-24T06:19:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64c806aed0 msgr2=0x7f64c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64c806aed0 0x7f64c806d390 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f64dc14bbc0 tx=0x7f64d0009040 comp rx=0 tx=0).stop 2023-11-24T06:19:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc06fa40 msgr2=0x7f64dc159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc06fa40 0x7f64dc159ab0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009bc0 tx=0x7f64cc004750 comp rx=0 tx=0).stop 2023-11-24T06:19:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 shutdown_connections 2023-11-24T06:19:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64dc0afc10 0x7f64dc146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc06fa40 0x7f64dc159ab0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64c806aed0 0x7f64c806d390 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 --2- 172.21.15.89:0/342619902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc0746a0 0x7f64dc145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 >> 172.21.15.89:0/342619902 conn(0x7f64dc06aff0 msgr2=0x7f64dc156510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 shutdown_connections 2023-11-24T06:19:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.312+0000 7f64e2631700 1 -- 172.21.15.89:0/342619902 wait complete. 2023-11-24T06:19:38.327 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-11-24T06:19:38.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.447+0000 7f861c8dc700 1 Processor -- start 2023-11-24T06:19:38.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.447+0000 7f861c8dc700 1 -- start start 2023-11-24T06:19:38.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.448+0000 7f861c8dc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f86180d2240 0x7f86180b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.448+0000 7f861c8dc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f861815e1b0 0x7f86180cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.448+0000 7f861c8dc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86180b1d40 con 0x7f861815ddd0 2023-11-24T06:19:38.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.448+0000 7f861c8dc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86180b28c0 con 0x7f86180d2240 2023-11-24T06:19:38.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.448+0000 7f861c8dc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861806c000 con 0x7f861815e1b0 2023-11-24T06:19:38.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.448+0000 7f861659c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f861815ddd0 0x7f86180d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40188/0 (socket says 172.21.15.89:40188) 2023-11-24T06:19:38.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.448+0000 7f861659c700 1 -- 172.21.15.89:0/1337712647 learned_addr learned my addr 172.21.15.89:0/1337712647 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:38.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8615d9b700 1 --2- 172.21.15.89:0/1337712647 >> v2:172.21.15.123:3300/0 conn(0x7f86180d2240 0x7f86180b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8616d9d700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f861815e1b0 0x7f86180cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504882304 0 0) 0x7f86180b1d40 con 0x7f861815ddd0 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86040035b0 con 0x7f861815ddd0 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3252531768 0 0) 0x7f86040035b0 con 0x7f861815ddd0 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f861815e1b0 msgr2=0x7f86180cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f861815e1b0 0x7f86180cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 >> v2:172.21.15.123:3300/0 conn(0x7f86180d2240 msgr2=0x7f86180b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 --2- 172.21.15.89:0/1337712647 >> v2:172.21.15.123:3300/0 conn(0x7f86180d2240 0x7f86180b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86180b1f90 con 0x7f861815ddd0 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f8608001670 con 0x7f861815ddd0 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 >> v1:172.21.15.89:6789/0 conn(0x7f861815ddd0 legacy=0x7f86180d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8604003a00 0x7f8604003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604004850 0x7f8604004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8604005210 0x7f860400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86180b28c0 con 0x7f8604005210 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861806c000 con 0x7f8604003a00 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860400c030 con 0x7f8604004850 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8616d9d700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f861815e1b0 0x7f86180cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8615d9b700 1 --2- 172.21.15.89:0/1337712647 >> v2:172.21.15.123:3300/0 conn(0x7f86180d2240 0x7f86180b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8616d9d700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8604005210 0x7f860400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f861659c700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8604003a00 0x7f8604003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8615d9b700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604004850 0x7f8604004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8616d9d700 1 -- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604004850 msgr2=0x7f8604004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8616d9d700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604004850 0x7f8604004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8616d9d700 1 -- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8604003a00 msgr2=0x7f8604003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8616d9d700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8604003a00 0x7f8604003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.449+0000 7f8616d9d700 1 -- 172.21.15.89:0/1337712647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040035b0 con 0x7f8604005210 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f8616d9d700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8604005210 0x7f860400ba10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f861815e590 tx=0x7f8604005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc3ce98c354d20b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610003c20 con 0x7f8604005210 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861000f040 con 0x7f8604005210 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861559a700 1 -- 172.21.15.89:0/1337712647 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861000ea60 con 0x7f8604005210 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861c8dc700 1 -- 172.21.15.89:0/1337712647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8604005210 msgr2=0x7f860400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861c8dc700 1 --2- 172.21.15.89:0/1337712647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8604005210 0x7f860400ba10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f861815e590 tx=0x7f8604005880 comp rx=0 tx=0).stop 2023-11-24T06:19:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f8616d9d700 1 -- 172.21.15.89:0/1337712647 reap_dead start 2023-11-24T06:19:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861c8dc700 1 -- 172.21.15.89:0/1337712647 shutdown_connections 2023-11-24T06:19:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861c8dc700 1 -- 172.21.15.89:0/1337712647 >> 172.21.15.89:0/1337712647 conn(0x7f86180731a0 msgr2=0x7f861809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861c8dc700 1 -- 172.21.15.89:0/1337712647 shutdown_connections 2023-11-24T06:19:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.450+0000 7f861c8dc700 1 -- 172.21.15.89:0/1337712647 wait complete. 2023-11-24T06:19:38.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.452+0000 7f861c8dc700 1 Processor -- start 2023-11-24T06:19:38.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.452+0000 7f861c8dc700 1 -- start start 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.452+0000 7f861c8dc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604005210 0x7f861814a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.452+0000 7f861c8dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861814da20 0x7f86181f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.452+0000 7f861c8dc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f861815f6f0 0x7f86180b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.452+0000 7f861c8dc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861814adb0 con 0x7f861814da20 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.452+0000 7f861c8dc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86181586a0 con 0x7f861815f6f0 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.452+0000 7f861c8dc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8618158940 con 0x7f8604005210 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861814da20 0x7f86181f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861814da20 0x7f86181f3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36698/0 (socket says 172.21.15.89:36698) 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 -- 172.21.15.89:0/4048437094 learned_addr learned my addr 172.21.15.89:0/4048437094 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f861659c700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604005210 0x7f861814a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8616d9d700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f861815f6f0 0x7f86180b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 -- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604005210 msgr2=0x7f861814a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604005210 0x7f861814a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 -- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f861815f6f0 msgr2=0x7f86180b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f861815f6f0 0x7f86180b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 -- 172.21.15.89:0/4048437094 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86180b2ed0 con 0x7f861814da20 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8616d9d700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f861815f6f0 0x7f86180b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f8615d9b700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861814da20 0x7f86181f3160 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8600011c20 tx=0x7f8600011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f860effd700 1 -- 172.21.15.89:0/4048437094 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860000fab0 con 0x7f861814da20 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f860effd700 1 -- 172.21.15.89:0/4048437094 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860000fc10 con 0x7f861814da20 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8618158c20 con 0x7f861814da20 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f860effd700 1 -- 172.21.15.89:0/4048437094 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600004020 con 0x7f861814da20 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.453+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86180b69b0 con 0x7f861814da20 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.454+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8618067f60 con 0x7f861814da20 2023-11-24T06:19:38.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.454+0000 7f860effd700 1 -- 172.21.15.89:0/4048437094 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8600004180 con 0x7f861814da20 2023-11-24T06:19:38.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.455+0000 7f860effd700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85f806aed0 0x7f85f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.455+0000 7f860effd700 1 -- 172.21.15.89:0/4048437094 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f86000898b0 con 0x7f861814da20 2023-11-24T06:19:38.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.455+0000 7f861659c700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85f806aed0 0x7f85f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.455+0000 7f861659c700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85f806aed0 0x7f85f806d390 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8608001550 tx=0x7f8608000f80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.459+0000 7f860effd700 1 -- 172.21.15.89:0/4048437094 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8600054d40 con 0x7f861814da20 2023-11-24T06:19:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.730+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f86180b7c60 con 0x7f861814da20 2023-11-24T06:19:38.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.732+0000 7f860effd700 1 -- 172.21.15.89:0/4048437094 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8600022070 con 0x7f861814da20 2023-11-24T06:19:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85f806aed0 msgr2=0x7f85f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85f806aed0 0x7f85f806d390 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8608001550 tx=0x7f8608000f80 comp rx=0 tx=0).stop 2023-11-24T06:19:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861814da20 msgr2=0x7f86181f3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861814da20 0x7f86181f3160 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8600011c20 tx=0x7f8600011f30 comp rx=0 tx=0).stop 2023-11-24T06:19:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 shutdown_connections 2023-11-24T06:19:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85f806aed0 0x7f85f806d390 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f861815f6f0 0x7f86180b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8604005210 0x7f861814a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 --2- 172.21.15.89:0/4048437094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861814da20 0x7f86181f3160 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 >> 172.21.15.89:0/4048437094 conn(0x7f86180731a0 msgr2=0x7f861806f400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:38.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 shutdown_connections 2023-11-24T06:19:38.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.734+0000 7f861c8dc700 1 -- 172.21.15.89:0/4048437094 wait complete. 2023-11-24T06:19:38.748 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-11-24T06:19:38.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.867+0000 7efeab349700 1 Processor -- start 2023-11-24T06:19:38.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.867+0000 7efeab349700 1 -- start start 2023-11-24T06:19:38.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efeab349700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea415bdd0 0x7efea41604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efeab349700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efea40d1c70 0x7efea40d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efeab349700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea40b22d0 con 0x7efea40b3370 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efeab349700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea40b2f50 con 0x7efea40d1c70 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efeab349700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea40d2640 con 0x7efea415bdd0 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea98e6700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efea40d1c70 0x7efea40d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea98e6700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efea40d1c70 0x7efea40d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36710/0 (socket says 172.21.15.89:36710) 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea98e6700 1 -- 172.21.15.89:0/459614273 learned_addr learned my addr 172.21.15.89:0/459614273 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea90e5700 1 --2- 172.21.15.89:0/459614273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea415bdd0 0x7efea41604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea98e6700 1 -- 172.21.15.89:0/459614273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea415bdd0 msgr2=0x7efea41604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea98e6700 1 --2- 172.21.15.89:0/459614273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea415bdd0 0x7efea41604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea98e6700 1 -- 172.21.15.89:0/459614273 >> v1:172.21.15.123:6789/0 conn(0x7efea40b3370 legacy=0x7efea40b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea98e6700 1 -- 172.21.15.89:0/459614273 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea40b25b0 con 0x7efea40d1c70 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.868+0000 7efea98e6700 1 --2- 172.21.15.89:0/459614273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea40d1c70 0x7efea40d2100 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009e50 tx=0x7efe9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d7ac8aa3a68cd1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efe9bfff700 1 -- 172.21.15.89:0/459614273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00e070 con 0x7efea40d1c70 2023-11-24T06:19:38.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efe9bfff700 1 -- 172.21.15.89:0/459614273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c003a20 con 0x7efea40d1c70 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efe9bfff700 1 -- 172.21.15.89:0/459614273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c003dd0 con 0x7efea40d1c70 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efeab349700 1 -- 172.21.15.89:0/459614273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea40d1c70 msgr2=0x7efea40d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efeab349700 1 --2- 172.21.15.89:0/459614273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea40d1c70 0x7efea40d2100 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009e50 tx=0x7efe9c00b330 comp rx=0 tx=0).stop 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efeab349700 1 -- 172.21.15.89:0/459614273 shutdown_connections 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efeab349700 1 --2- 172.21.15.89:0/459614273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea40d1c70 0x7efea40d2100 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efeab349700 1 --2- 172.21.15.89:0/459614273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea415bdd0 0x7efea41604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efeab349700 1 -- 172.21.15.89:0/459614273 >> 172.21.15.89:0/459614273 conn(0x7efea40731a0 msgr2=0x7efea409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efeab349700 1 -- 172.21.15.89:0/459614273 shutdown_connections 2023-11-24T06:19:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.869+0000 7efeab349700 1 -- 172.21.15.89:0/459614273 wait complete. 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.870+0000 7efeab349700 1 Processor -- start 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.870+0000 7efeab349700 1 -- start start 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efeab349700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea40b3370 0x7efea40c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efeab349700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea40d1c70 0x7efea4159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efeab349700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea415bdd0 0x7efea414fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efeab349700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea40cbb50 con 0x7efea415bdd0 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efeab349700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea40caab0 con 0x7efea40d1c70 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efeab349700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea40cad90 con 0x7efea40b3370 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efea98e6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea415bdd0 0x7efea414fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efea98e6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea415bdd0 0x7efea414fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36716/0 (socket says 172.21.15.89:36716) 2023-11-24T06:19:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efea98e6700 1 -- 172.21.15.89:0/600324200 learned_addr learned my addr 172.21.15.89:0/600324200 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efea90e5700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea40b3370 0x7efea40c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efea88e4700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea40d1c70 0x7efea4159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.871+0000 7efea98e6700 1 -- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea40b3370 msgr2=0x7efea40c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efea98e6700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea40b3370 0x7efea40c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efea98e6700 1 -- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea40d1c70 msgr2=0x7efea4159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efea98e6700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea40d1c70 0x7efea4159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efea98e6700 1 -- 172.21.15.89:0/600324200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c009b30 con 0x7efea415bdd0 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efea88e4700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea40d1c70 0x7efea4159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efea98e6700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea415bdd0 0x7efea414fe60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efea40716d0 tx=0x7efe9c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efe99ffb700 1 -- 172.21.15.89:0/600324200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c004440 con 0x7efea415bdd0 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efe99ffb700 1 -- 172.21.15.89:0/600324200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c0045a0 con 0x7efea415bdd0 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea40cb070 con 0x7efea415bdd0 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efe99ffb700 1 -- 172.21.15.89:0/600324200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c004950 con 0x7efea415bdd0 2023-11-24T06:19:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.872+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea40cb630 con 0x7efea415bdd0 2023-11-24T06:19:38.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.873+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea40cb7f0 con 0x7efea415bdd0 2023-11-24T06:19:38.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.873+0000 7efe99ffb700 1 -- 172.21.15.89:0/600324200 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efe9c012430 con 0x7efea415bdd0 2023-11-24T06:19:38.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.874+0000 7efe99ffb700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efe8806aed0 0x7efe8806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:38.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.874+0000 7efe99ffb700 1 -- 172.21.15.89:0/600324200 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efe9c08a6b0 con 0x7efea415bdd0 2023-11-24T06:19:38.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.874+0000 7efea90e5700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efe8806aed0 0x7efe8806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:38.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.874+0000 7efea90e5700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efe8806aed0 0x7efe8806d390 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efea41f5100 tx=0x7efe94008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:38.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:38.878+0000 7efe99ffb700 1 -- 172.21.15.89:0/600324200 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe9c0595d0 con 0x7efea415bdd0 2023-11-24T06:19:39.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.143+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7efea40d01b0 con 0x7efea415bdd0 2023-11-24T06:19:39.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.146+0000 7efe99ffb700 1 -- 172.21.15.89:0/600324200 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7efe9c059160 con 0x7efea415bdd0 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.148+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efe8806aed0 msgr2=0x7efe8806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.148+0000 7efeab349700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efe8806aed0 0x7efe8806d390 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efea41f5100 tx=0x7efe94008040 comp rx=0 tx=0).stop 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.148+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea415bdd0 msgr2=0x7efea414fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.148+0000 7efeab349700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea415bdd0 0x7efea414fe60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efea40716d0 tx=0x7efe9c004ad0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.149+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 shutdown_connections 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.149+0000 7efeab349700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea415bdd0 0x7efea414fe60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.149+0000 7efeab349700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efea40b3370 0x7efea40c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.149+0000 7efeab349700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efe8806aed0 0x7efe8806d390 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.149+0000 7efeab349700 1 --2- 172.21.15.89:0/600324200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efea40d1c70 0x7efea4159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.149+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 >> 172.21.15.89:0/600324200 conn(0x7efea40731a0 msgr2=0x7efea40cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:39.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.149+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 shutdown_connections 2023-11-24T06:19:39.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.149+0000 7efeab349700 1 -- 172.21.15.89:0/600324200 wait complete. 2023-11-24T06:19:39.162 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-11-24T06:19:39.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.282+0000 7f7628d73700 1 Processor -- start 2023-11-24T06:19:39.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.282+0000 7f7628d73700 1 -- start start 2023-11-24T06:19:39.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7628d73700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f76240d2380 0x7f76241604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7628d73700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76240b22d0 con 0x7f76240b3370 2023-11-24T06:19:39.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7628d73700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76240b25b0 con 0x7f762415bdd0 2023-11-24T06:19:39.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7628d73700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76240b2f50 con 0x7f76240d2380 2023-11-24T06:19:39.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7621d9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f76240b3370 0x7f76240d1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40226/0 (socket says 172.21.15.89:40226) 2023-11-24T06:19:39.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7621d9b700 1 -- 172.21.15.89:0/2289283053 learned_addr learned my addr 172.21.15.89:0/2289283053 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:39.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7622d9d700 1 --2- 172.21.15.89:0/2289283053 >> v2:172.21.15.89:3300/0 conn(0x7f76240d2380 0x7f76241604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f762159a700 1 -- 172.21.15.89:0/2289283053 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120704638 0 0) 0x7f76240b22d0 con 0x7f76240b3370 2023-11-24T06:19:39.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f762159a700 1 -- 172.21.15.89:0/2289283053 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f760c0035b0 con 0x7f76240b3370 2023-11-24T06:19:39.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7622d9d700 1 -- 172.21.15.89:0/2289283053 >> v1:172.21.15.123:6789/0 conn(0x7f762415bdd0 legacy=0x7f76240b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7622d9d700 1 -- 172.21.15.89:0/2289283053 >> v1:172.21.15.89:6789/0 conn(0x7f76240b3370 legacy=0x7f76240d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.283+0000 7f7622d9d700 1 -- 172.21.15.89:0/2289283053 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76240b2890 con 0x7f76240d2380 2023-11-24T06:19:39.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f7622d9d700 1 --2- 172.21.15.89:0/2289283053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240d2380 0x7f76241604e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f761c009d90 tx=0x7f761c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216047ab6a81ddf6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:39.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f762159a700 1 -- 172.21.15.89:0/2289283053 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c00e070 con 0x7f76240d2380 2023-11-24T06:19:39.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f762159a700 1 -- 172.21.15.89:0/2289283053 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c00bb20 con 0x7f76240d2380 2023-11-24T06:19:39.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f762159a700 1 -- 172.21.15.89:0/2289283053 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c012430 con 0x7f76240d2380 2023-11-24T06:19:39.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f7628d73700 1 -- 172.21.15.89:0/2289283053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240d2380 msgr2=0x7f76241604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f7628d73700 1 --2- 172.21.15.89:0/2289283053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240d2380 0x7f76241604e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f761c009d90 tx=0x7f761c00b330 comp rx=0 tx=0).stop 2023-11-24T06:19:39.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f7628d73700 1 -- 172.21.15.89:0/2289283053 shutdown_connections 2023-11-24T06:19:39.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f7628d73700 1 --2- 172.21.15.89:0/2289283053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240d2380 0x7f76241604e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f7628d73700 1 -- 172.21.15.89:0/2289283053 >> 172.21.15.89:0/2289283053 conn(0x7f76240731a0 msgr2=0x7f762409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:39.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f7628d73700 1 -- 172.21.15.89:0/2289283053 shutdown_connections 2023-11-24T06:19:39.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.284+0000 7f7628d73700 1 -- 172.21.15.89:0/2289283053 wait complete. 2023-11-24T06:19:39.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.286+0000 7f7628d73700 1 Processor -- start 2023-11-24T06:19:39.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.286+0000 7f7628d73700 1 -- start start 2023-11-24T06:19:39.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.286+0000 7f7628d73700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240b3370 0x7f76240afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.286+0000 7f7628d73700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76240d2380 0x7f76240cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7628d73700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f762415bdd0 0x7f7624148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7628d73700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762406aff0 con 0x7f76240b3370 2023-11-24T06:19:39.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7628d73700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762415b4d0 con 0x7f762415bdd0 2023-11-24T06:19:39.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7628d73700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762415b7b0 con 0x7f76240d2380 2023-11-24T06:19:39.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240b3370 0x7f76240afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240b3370 0x7f76240afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36726/0 (socket says 172.21.15.89:36726) 2023-11-24T06:19:39.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 -- 172.21.15.89:0/2843402356 learned_addr learned my addr 172.21.15.89:0/2843402356 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:39.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7621d9b700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76240d2380 0x7f76240cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7622d9d700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f762415bdd0 0x7f7624148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 -- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76240d2380 msgr2=0x7f76240cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76240d2380 0x7f76240cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 -- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f762415bdd0 msgr2=0x7f7624148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f762415bdd0 0x7f7624148050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 -- 172.21.15.89:0/2843402356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c009a70 con 0x7f76240b3370 2023-11-24T06:19:39.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7622d9d700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f762415bdd0 0x7f7624148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:39.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f762259c700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240b3370 0x7f76240afe70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f761800c450 tx=0x7f761800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:39.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7612ffd700 1 -- 172.21.15.89:0/2843402356 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180052d0 con 0x7f76240b3370 2023-11-24T06:19:39.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7612ffd700 1 -- 172.21.15.89:0/2843402356 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7618005430 con 0x7f76240b3370 2023-11-24T06:19:39.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762406b200 con 0x7f76240b3370 2023-11-24T06:19:39.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7612ffd700 1 -- 172.21.15.89:0/2843402356 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180038c0 con 0x7f76240b3370 2023-11-24T06:19:39.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.287+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762406b7c0 con 0x7f76240b3370 2023-11-24T06:19:39.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.288+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762406b980 con 0x7f76240b3370 2023-11-24T06:19:39.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.289+0000 7f7612ffd700 1 -- 172.21.15.89:0/2843402356 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7618003a20 con 0x7f76240b3370 2023-11-24T06:19:39.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.289+0000 7f7612ffd700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f760406aed0 0x7f760406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.289+0000 7f7612ffd700 1 -- 172.21.15.89:0/2843402356 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7618088590 con 0x7f76240b3370 2023-11-24T06:19:39.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.289+0000 7f7621d9b700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f760406aed0 0x7f760406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.289+0000 7f7621d9b700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f760406aed0 0x7f760406d390 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f76241580f0 tx=0x7f7614009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:39.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.294+0000 7f7612ffd700 1 -- 172.21.15.89:0/2843402356 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7618053990 con 0x7f76240b3370 2023-11-24T06:19:39.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.562+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f7624067f60 con 0x7f76240b3370 2023-11-24T06:19:39.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.566+0000 7f7612ffd700 1 -- 172.21.15.89:0/2843402356 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7618056fb0 con 0x7f76240b3370 2023-11-24T06:19:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f760406aed0 msgr2=0x7f760406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f760406aed0 0x7f760406d390 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f76241580f0 tx=0x7f7614009040 comp rx=0 tx=0).stop 2023-11-24T06:19:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240b3370 msgr2=0x7f76240afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240b3370 0x7f76240afe70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f761800c450 tx=0x7f761800c810 comp rx=0 tx=0).stop 2023-11-24T06:19:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 shutdown_connections 2023-11-24T06:19:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f760406aed0 0x7f760406d390 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f762415bdd0 0x7f7624148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240b3370 0x7f76240afe70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 --2- 172.21.15.89:0/2843402356 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76240d2380 0x7f76240cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 >> 172.21.15.89:0/2843402356 conn(0x7f76240731a0 msgr2=0x7f762415f940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:39.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 shutdown_connections 2023-11-24T06:19:39.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.568+0000 7f7628d73700 1 -- 172.21.15.89:0/2843402356 wait complete. 2023-11-24T06:19:39.582 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-11-24T06:19:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.701+0000 7f8ae46f2700 1 Processor -- start 2023-11-24T06:19:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.701+0000 7f8ae46f2700 1 -- start start 2023-11-24T06:19:39.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.702+0000 7f8ae46f2700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8adc15e080 0x7f8adc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.702+0000 7f8ae46f2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8adc06c710 0x7f8adc0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.702+0000 7f8ae46f2700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc0ae430 con 0x7f8adc0af4d0 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.702+0000 7f8ae46f2700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8adc0af0b0 con 0x7f8adc15e080 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.702+0000 7f8ae46f2700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8adc0b05a0 con 0x7f8adc06c710 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.702+0000 7f8ae248e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8adc15e080 0x7f8adc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.702+0000 7f8ae248e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8adc15e080 0x7f8adc0b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36734/0 (socket says 172.21.15.89:36734) 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.702+0000 7f8ae248e700 1 -- 172.21.15.89:0/3021263107 learned_addr learned my addr 172.21.15.89:0/3021263107 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae248e700 1 -- 172.21.15.89:0/3021263107 >> v2:172.21.15.123:3300/0 conn(0x7f8adc06c710 msgr2=0x7f8adc0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae248e700 1 --2- 172.21.15.89:0/3021263107 >> v2:172.21.15.123:3300/0 conn(0x7f8adc06c710 0x7f8adc0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae248e700 1 -- 172.21.15.89:0/3021263107 >> v1:172.21.15.123:6789/0 conn(0x7f8adc0af4d0 legacy=0x7f8adc06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae248e700 1 -- 172.21.15.89:0/3021263107 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc0ae710 con 0x7f8adc15e080 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae248e700 1 --2- 172.21.15.89:0/3021263107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0b5050 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8adc06e060 tx=0x7f8acc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61ca2198b3d53fc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae148c700 1 -- 172.21.15.89:0/3021263107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc004230 con 0x7f8adc15e080 2023-11-24T06:19:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae148c700 1 -- 172.21.15.89:0/3021263107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc004390 con 0x7f8adc15e080 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae148c700 1 -- 172.21.15.89:0/3021263107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc0036a0 con 0x7f8adc15e080 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae46f2700 1 -- 172.21.15.89:0/3021263107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 msgr2=0x7f8adc0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/3021263107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0b5050 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8adc06e060 tx=0x7f8acc009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae46f2700 1 -- 172.21.15.89:0/3021263107 shutdown_connections 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/3021263107 >> v2:172.21.15.123:3300/0 conn(0x7f8adc06c710 0x7f8adc0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/3021263107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0b5050 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.703+0000 7f8ae46f2700 1 -- 172.21.15.89:0/3021263107 >> 172.21.15.89:0/3021263107 conn(0x7f8adc071b90 msgr2=0x7f8adc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.704+0000 7f8ae46f2700 1 -- 172.21.15.89:0/3021263107 shutdown_connections 2023-11-24T06:19:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.704+0000 7f8ae46f2700 1 -- 172.21.15.89:0/3021263107 wait complete. 2023-11-24T06:19:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.705+0000 7f8ae46f2700 1 Processor -- start 2023-11-24T06:19:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.705+0000 7f8ae46f2700 1 -- start start 2023-11-24T06:19:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.705+0000 7f8ae46f2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8adc06c710 0x7f8adc14c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.705+0000 7f8ae46f2700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8adc0af4d0 0x7f8adc14c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.705+0000 7f8ae46f2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.705+0000 7f8ae46f2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0c7660 con 0x7f8adc15e080 2023-11-24T06:19:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.705+0000 7f8ae46f2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0cfa20 con 0x7f8adc06c710 2023-11-24T06:19:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.705+0000 7f8ae46f2700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0c6cc0 con 0x7f8adc0af4d0 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0cc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36744/0 (socket says 172.21.15.89:36744) 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 -- 172.21.15.89:0/2290553990 learned_addr learned my addr 172.21.15.89:0/2290553990 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae248e700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8adc06c710 0x7f8adc14c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae1c8d700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8adc0af4d0 0x7f8adc14c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 -- 172.21.15.89:0/2290553990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8adc0af4d0 msgr2=0x7f8adc14c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8adc0af4d0 0x7f8adc14c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 -- 172.21.15.89:0/2290553990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8adc06c710 msgr2=0x7f8adc14c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8adc06c710 0x7f8adc14c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 -- 172.21.15.89:0/2290553990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc009550 con 0x7f8adc15e080 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae2c8f700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0cc200 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8ad400b9e0 tx=0x7f8ad400bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ad2ffd700 1 -- 172.21.15.89:0/2290553990 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad400d8e0 con 0x7f8adc15e080 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ad2ffd700 1 -- 172.21.15.89:0/2290553990 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad400da60 con 0x7f8adc15e080 2023-11-24T06:19:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc0c6fa0 con 0x7f8adc15e080 2023-11-24T06:19:39.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ad2ffd700 1 -- 172.21.15.89:0/2290553990 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad400de10 con 0x7f8adc15e080 2023-11-24T06:19:39.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.706+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc14cf30 con 0x7f8adc15e080 2023-11-24T06:19:39.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.707+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc0c78d0 con 0x7f8adc15e080 2023-11-24T06:19:39.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.708+0000 7f8ad2ffd700 1 -- 172.21.15.89:0/2290553990 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8ad4015420 con 0x7f8adc15e080 2023-11-24T06:19:39.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.708+0000 7f8ad2ffd700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8ac006aed0 0x7f8ac006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:39.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.708+0000 7f8ae248e700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8ac006aed0 0x7f8ac006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:39.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.708+0000 7f8ad2ffd700 1 -- 172.21.15.89:0/2290553990 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8ad4011070 con 0x7f8adc15e080 2023-11-24T06:19:39.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.708+0000 7f8ae248e700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8ac006aed0 0x7f8ac006d390 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009130 tx=0x7f8acc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:39.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.713+0000 7f8ad2ffd700 1 -- 172.21.15.89:0/2290553990 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ad404ee90 con 0x7f8adc15e080 2023-11-24T06:19:39.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.982+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f8adc0753f0 con 0x7f8adc15e080 2023-11-24T06:19:39.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.986+0000 7f8ad2ffd700 1 -- 172.21.15.89:0/2290553990 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8ad4054540 con 0x7f8adc15e080 2023-11-24T06:19:39.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8ac006aed0 msgr2=0x7f8ac006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8ac006aed0 0x7f8ac006d390 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009130 tx=0x7f8acc008030 comp rx=0 tx=0).stop 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 msgr2=0x7f8adc0cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0cc200 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8ad400b9e0 tx=0x7f8ad400bcf0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 shutdown_connections 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc15e080 0x7f8adc0cc200 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8ac006aed0 0x7f8ac006d390 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8adc06c710 0x7f8adc14c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 --2- 172.21.15.89:0/2290553990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8adc0af4d0 0x7f8adc14c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 >> 172.21.15.89:0/2290553990 conn(0x7f8adc071b90 msgr2=0x7f8adc160470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 shutdown_connections 2023-11-24T06:19:39.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:39.988+0000 7f8ae46f2700 1 -- 172.21.15.89:0/2290553990 wait complete. 2023-11-24T06:19:40.002 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-11-24T06:19:40.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.122+0000 7f4ed571a700 1 Processor -- start 2023-11-24T06:19:40.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.122+0000 7f4ed571a700 1 -- start start 2023-11-24T06:19:40.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.122+0000 7f4ed571a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4ed0145760 0x7f4ed006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.122+0000 7f4ed571a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed015ffb0 0x7f4ed00cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.122+0000 7f4ed571a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed006ee70 con 0x7f4ed006df20 2023-11-24T06:19:40.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.122+0000 7f4ed571a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed006d4d0 con 0x7f4ed0145760 2023-11-24T06:19:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.122+0000 7f4ed571a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00cc5e0 con 0x7f4ed015ffb0 2023-11-24T06:19:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4ece7fc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4ed006df20 0x7f4ed015f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40252/0 (socket says 172.21.15.89:40252) 2023-11-24T06:19:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4ece7fc700 1 -- 172.21.15.89:0/2309851873 learned_addr learned my addr 172.21.15.89:0/2309851873 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4ecf7fe700 1 --2- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed015ffb0 0x7f4ed00cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4eceffd700 1 --2- 172.21.15.89:0/2309851873 >> v2:172.21.15.123:3300/0 conn(0x7f4ed0145760 0x7f4ed006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4ecdffb700 1 -- 172.21.15.89:0/2309851873 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385745662 0 0) 0x7f4ed006ee70 con 0x7f4ed006df20 2023-11-24T06:19:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4ecdffb700 1 -- 172.21.15.89:0/2309851873 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ebc0035b0 con 0x7f4ed006df20 2023-11-24T06:19:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4eceffd700 1 -- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed015ffb0 msgr2=0x7f4ed00cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4eceffd700 1 --2- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed015ffb0 0x7f4ed00cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4eceffd700 1 -- 172.21.15.89:0/2309851873 >> v1:172.21.15.89:6789/0 conn(0x7f4ed006df20 legacy=0x7f4ed015f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4eceffd700 1 -- 172.21.15.89:0/2309851873 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00cc8c0 con 0x7f4ed0145760 2023-11-24T06:19:40.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4ecf7fe700 1 --2- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed015ffb0 0x7f4ed00cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4eceffd700 1 --2- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 0x7f4ed006e930 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009840 tx=0x7f4ec0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38f3a862a59a1356 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.123+0000 7f4ecdffb700 1 -- 172.21.15.89:0/2309851873 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0011830 con 0x7f4ed0145760 2023-11-24T06:19:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ecdffb700 1 -- 172.21.15.89:0/2309851873 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec0011990 con 0x7f4ed0145760 2023-11-24T06:19:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ecdffb700 1 -- 172.21.15.89:0/2309851873 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0011d00 con 0x7f4ed0145760 2023-11-24T06:19:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ed571a700 1 -- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 msgr2=0x7f4ed006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ed571a700 1 --2- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 0x7f4ed006e930 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009840 tx=0x7f4ec0009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:40.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ed571a700 1 -- 172.21.15.89:0/2309851873 shutdown_connections 2023-11-24T06:19:40.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ed571a700 1 --2- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed015ffb0 0x7f4ed00cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ed571a700 1 --2- 172.21.15.89:0/2309851873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 0x7f4ed006e930 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ed571a700 1 -- 172.21.15.89:0/2309851873 >> 172.21.15.89:0/2309851873 conn(0x7f4ed00b1740 msgr2=0x7f4ed00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ed571a700 1 -- 172.21.15.89:0/2309851873 shutdown_connections 2023-11-24T06:19:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.124+0000 7f4ed571a700 1 -- 172.21.15.89:0/2309851873 wait complete. 2023-11-24T06:19:40.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.126+0000 7f4ed571a700 1 Processor -- start 2023-11-24T06:19:40.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.126+0000 7f4ed571a700 1 -- start start 2023-11-24T06:19:40.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.126+0000 7f4ed571a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed006df20 0x7f4ed01403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.126+0000 7f4ed571a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 0x7f4ed015b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.126+0000 7f4ed571a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ed015ffb0 0x7f4ed00af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.126+0000 7f4ed571a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0143c40 con 0x7f4ed015ffb0 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.126+0000 7f4ed571a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0143f20 con 0x7f4ed0145760 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.126+0000 7f4ed571a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0144200 con 0x7f4ed006df20 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ed015ffb0 0x7f4ed00af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4eceffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed006df20 0x7f4ed01403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ed015ffb0 0x7f4ed00af8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36756/0 (socket says 172.21.15.89:36756) 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 -- 172.21.15.89:0/3122763367 learned_addr learned my addr 172.21.15.89:0/3122763367 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ece7fc700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 0x7f4ed015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 -- 172.21.15.89:0/3122763367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed006df20 msgr2=0x7f4ed01403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed006df20 0x7f4ed01403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 -- 172.21.15.89:0/3122763367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 msgr2=0x7f4ed015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 0x7f4ed015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 -- 172.21.15.89:0/3122763367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec8009a70 con 0x7f4ed015ffb0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ecf7fe700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ed015ffb0 0x7f4ed00af8f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800efc0 tx=0x7f4ec800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ec77fe700 1 -- 172.21.15.89:0/3122763367 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800ca90 con 0x7f4ed015ffb0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ec77fe700 1 -- 172.21.15.89:0/3122763367 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec800f040 con 0x7f4ed015ffb0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec00094a0 con 0x7f4ed015ffb0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ec77fe700 1 -- 172.21.15.89:0/3122763367 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800ce30 con 0x7f4ed015ffb0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.127+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed01447c0 con 0x7f4ed015ffb0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.128+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed00751e0 con 0x7f4ed015ffb0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.129+0000 7f4ec77fe700 1 -- 172.21.15.89:0/3122763367 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4ec8014480 con 0x7f4ed015ffb0 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.129+0000 7f4ec77fe700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4eb006aed0 0x7f4eb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.129+0000 7f4ec77fe700 1 -- 172.21.15.89:0/3122763367 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4ec80898a0 con 0x7f4ed015ffb0 2023-11-24T06:19:40.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.129+0000 7f4eceffd700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4eb006aed0 0x7f4eb006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.129+0000 7f4eceffd700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4eb006aed0 0x7f4eb006d390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009fd0 tx=0x7f4ec0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.133+0000 7f4ec77fe700 1 -- 172.21.15.89:0/3122763367 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ec8054ca0 con 0x7f4ed015ffb0 2023-11-24T06:19:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.405+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f4ed0144ed0 con 0x7f4ed015ffb0 2023-11-24T06:19:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.409+0000 7f4ec77fe700 1 -- 172.21.15.89:0/3122763367 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4ec8009cd0 con 0x7f4ed015ffb0 2023-11-24T06:19:40.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4eb006aed0 msgr2=0x7f4eb006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4eb006aed0 0x7f4eb006d390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009fd0 tx=0x7f4ec0008030 comp rx=0 tx=0).stop 2023-11-24T06:19:40.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ed015ffb0 msgr2=0x7f4ed00af8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ed015ffb0 0x7f4ed00af8f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800efc0 tx=0x7f4ec800c3a0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 shutdown_connections 2023-11-24T06:19:40.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ed015ffb0 0x7f4ed00af8f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed006df20 0x7f4ed01403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4eb006aed0 0x7f4eb006d390 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 --2- 172.21.15.89:0/3122763367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed0145760 0x7f4ed015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 >> 172.21.15.89:0/3122763367 conn(0x7f4ed00b1740 msgr2=0x7f4ed00cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:40.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 shutdown_connections 2023-11-24T06:19:40.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.411+0000 7f4ed571a700 1 -- 172.21.15.89:0/3122763367 wait complete. 2023-11-24T06:19:40.425 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.543+0000 7f85e0ba4700 1 Processor -- start 2023-11-24T06:19:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.543+0000 7f85e0ba4700 1 -- start start 2023-11-24T06:19:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85e0ba4700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f85dc15bdd0 0x7f85dc1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85e0ba4700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85dc0b3370 0x7f85dc0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85e0ba4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0d1c70 0x7f85dc0d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85e0ba4700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc0b2f50 con 0x7f85dc0b3370 2023-11-24T06:19:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85e0ba4700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc0d2660 con 0x7f85dc15bdd0 2023-11-24T06:19:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85e0ba4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0b2030 con 0x7f85dc0d1c70 2023-11-24T06:19:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85dc0b3370 0x7f85dc0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85dc0b3370 0x7f85dc0b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36760/0 (socket says 172.21.15.89:36760) 2023-11-24T06:19:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 -- 172.21.15.89:0/1494477275 learned_addr learned my addr 172.21.15.89:0/1494477275 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 -- 172.21.15.89:0/1494477275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0d1c70 msgr2=0x7f85dc0d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85da59c700 1 --2- 172.21.15.89:0/1494477275 >> v2:172.21.15.123:3300/0 conn(0x7f85dc15bdd0 0x7f85dc1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85dad9d700 1 --2- 172.21.15.89:0/1494477275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0d1c70 0x7f85dc0d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 --2- 172.21.15.89:0/1494477275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0d1c70 0x7f85dc0d2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 -- 172.21.15.89:0/1494477275 >> v2:172.21.15.123:3300/0 conn(0x7f85dc15bdd0 msgr2=0x7f85dc1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 --2- 172.21.15.89:0/1494477275 >> v2:172.21.15.123:3300/0 conn(0x7f85dc15bdd0 0x7f85dc1604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 -- 172.21.15.89:0/1494477275 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc0b22d0 con 0x7f85dc0b3370 2023-11-24T06:19:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d9d9b700 1 --2- 172.21.15.89:0/1494477275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc0b3370 0x7f85dc0b1af0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f85dc06ec10 tx=0x7f85d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608529c80ae11710 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d959a700 1 -- 172.21.15.89:0/1494477275 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d000b870 con 0x7f85dc0b3370 2023-11-24T06:19:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d959a700 1 -- 172.21.15.89:0/1494477275 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d000b9d0 con 0x7f85dc0b3370 2023-11-24T06:19:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.544+0000 7f85d959a700 1 -- 172.21.15.89:0/1494477275 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d000bd60 con 0x7f85dc0b3370 2023-11-24T06:19:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 -- 172.21.15.89:0/1494477275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc0b3370 msgr2=0x7f85dc0b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/1494477275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc0b3370 0x7f85dc0b1af0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f85dc06ec10 tx=0x7f85d0009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 -- 172.21.15.89:0/1494477275 shutdown_connections 2023-11-24T06:19:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/1494477275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0d1c70 0x7f85dc0d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/1494477275 >> v2:172.21.15.123:3300/0 conn(0x7f85dc15bdd0 0x7f85dc1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/1494477275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc0b3370 0x7f85dc0b1af0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 -- 172.21.15.89:0/1494477275 >> 172.21.15.89:0/1494477275 conn(0x7f85dc0731a0 msgr2=0x7f85dc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 -- 172.21.15.89:0/1494477275 shutdown_connections 2023-11-24T06:19:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.545+0000 7f85e0ba4700 1 -- 172.21.15.89:0/1494477275 wait complete. 2023-11-24T06:19:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.546+0000 7f85e0ba4700 1 Processor -- start 2023-11-24T06:19:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.546+0000 7f85e0ba4700 1 -- start start 2023-11-24T06:19:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85e0ba4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0b3370 0x7f85dc0c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85e0ba4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85dc0d1c70 0x7f85dc14f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85e0ba4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc15bdd0 0x7f85dc15a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85e0ba4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0b4fa0 con 0x7f85dc15bdd0 2023-11-24T06:19:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85e0ba4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0c5fb0 con 0x7f85dc0d1c70 2023-11-24T06:19:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85e0ba4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0c6290 con 0x7f85dc0b3370 2023-11-24T06:19:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85dad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc15bdd0 0x7f85dc15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85dad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc15bdd0 0x7f85dc15a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36770/0 (socket says 172.21.15.89:36770) 2023-11-24T06:19:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85dad9d700 1 -- 172.21.15.89:0/2815470827 learned_addr learned my addr 172.21.15.89:0/2815470827 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85da59c700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0b3370 0x7f85dc0c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85d9d9b700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85dc0d1c70 0x7f85dc14f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85dad9d700 1 -- 172.21.15.89:0/2815470827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0b3370 msgr2=0x7f85dc0c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85dad9d700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0b3370 0x7f85dc0c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85dad9d700 1 -- 172.21.15.89:0/2815470827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85dc0d1c70 msgr2=0x7f85dc14f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85dad9d700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85dc0d1c70 0x7f85dc14f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.547+0000 7f85dad9d700 1 -- 172.21.15.89:0/2815470827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d0009550 con 0x7f85dc15bdd0 2023-11-24T06:19:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.548+0000 7f85dad9d700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc15bdd0 0x7f85dc15a170 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f85d400ccf0 tx=0x7f85d400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.548+0000 7f85caffd700 1 -- 172.21.15.89:0/2815470827 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d4003a20 con 0x7f85dc15bdd0 2023-11-24T06:19:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.548+0000 7f85caffd700 1 -- 172.21.15.89:0/2815470827 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d4003b80 con 0x7f85dc15bdd0 2023-11-24T06:19:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.548+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc0c6570 con 0x7f85dc15bdd0 2023-11-24T06:19:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.548+0000 7f85caffd700 1 -- 172.21.15.89:0/2815470827 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d4004020 con 0x7f85dc15bdd0 2023-11-24T06:19:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.548+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc0c5ca0 con 0x7f85dc15bdd0 2023-11-24T06:19:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.549+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85dc0c6a90 con 0x7f85dc15bdd0 2023-11-24T06:19:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.549+0000 7f85caffd700 1 -- 172.21.15.89:0/2815470827 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f85d4004180 con 0x7f85dc15bdd0 2023-11-24T06:19:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.550+0000 7f85caffd700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85bc06aed0 0x7f85bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.550+0000 7f85da59c700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85bc06aed0 0x7f85bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.550+0000 7f85caffd700 1 -- 172.21.15.89:0/2815470827 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f85d4012070 con 0x7f85dc15bdd0 2023-11-24T06:19:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.550+0000 7f85da59c700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85bc06aed0 0x7f85bc06d390 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00b840 tx=0x7f85cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.554+0000 7f85caffd700 1 -- 172.21.15.89:0/2815470827 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85d4054e90 con 0x7f85dc15bdd0 2023-11-24T06:19:40.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.745+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85dc067f60 con 0x7f85dc15bdd0 2023-11-24T06:19:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.746+0000 7f85caffd700 1 -- 172.21.15.89:0/2815470827 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12821 (secure 0 0 0) 0x7f85d40584b0 con 0x7f85dc15bdd0 2023-11-24T06:19:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:40.748 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":30,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:19:37.468568+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85bc06aed0 msgr2=0x7f85bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85bc06aed0 0x7f85bc06d390 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00b840 tx=0x7f85cc008040 comp rx=0 tx=0).stop 2023-11-24T06:19:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc15bdd0 msgr2=0x7f85dc15a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc15bdd0 0x7f85dc15a170 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f85d400ccf0 tx=0x7f85d400a570 comp rx=0 tx=0).stop 2023-11-24T06:19:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 shutdown_connections 2023-11-24T06:19:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc15bdd0 0x7f85dc15a170 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85dc0b3370 0x7f85dc0c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f85bc06aed0 0x7f85bc06d390 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.748+0000 7f85e0ba4700 1 --2- 172.21.15.89:0/2815470827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85dc0d1c70 0x7f85dc14f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.749+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 >> 172.21.15.89:0/2815470827 conn(0x7f85dc0731a0 msgr2=0x7f85dc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:40.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.749+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 shutdown_connections 2023-11-24T06:19:40.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.749+0000 7f85e0ba4700 1 -- 172.21.15.89:0/2815470827 wait complete. 2023-11-24T06:19:40.764 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:19:40.764 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:19:40.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.885+0000 7f7354a02700 1 Processor -- start 2023-11-24T06:19:40.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.885+0000 7f7354a02700 1 -- start start 2023-11-24T06:19:40.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f7354a02700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735015e160 0x7f73500d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f7354a02700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73500d20d0 0x7f73500b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f7354a02700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73500cfe20 con 0x7f73500b1e00 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f7354a02700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73500d2550 con 0x7f73500d20d0 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f7354a02700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73500b2bb0 con 0x7f735015e160 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734dd9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73500d20d0 0x7f73500b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734ed9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f73500b1e00 0x7f73500ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40278/0 (socket says 172.21.15.89:40278) 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734ed9d700 1 -- 172.21.15.89:0/115153622 learned_addr learned my addr 172.21.15.89:0/115153622 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734e59c700 1 --2- 172.21.15.89:0/115153622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735015e160 0x7f73500d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734dd9b700 1 -- 172.21.15.89:0/115153622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735015e160 msgr2=0x7f73500d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734dd9b700 1 --2- 172.21.15.89:0/115153622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735015e160 0x7f73500d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734dd9b700 1 -- 172.21.15.89:0/115153622 >> v1:172.21.15.89:6789/0 conn(0x7f73500b1e00 legacy=0x7f73500ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734dd9b700 1 -- 172.21.15.89:0/115153622 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735015e540 con 0x7f73500d20d0 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734d59a700 1 -- 172.21.15.89:0/115153622 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249282373 0 0) 0x7f73500cfe20 con 0x7f73500b1e00 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734e59c700 1 --2- 172.21.15.89:0/115153622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735015e160 0x7f73500d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:40.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.886+0000 7f734dd9b700 1 --2- 172.21.15.89:0/115153622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73500d20d0 0x7f73500b18c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7344009840 tx=0x7f7344009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652b37d05c0f4224 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f734d59a700 1 -- 172.21.15.89:0/115153622 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344011830 con 0x7f73500d20d0 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f734d59a700 1 -- 172.21.15.89:0/115153622 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7344011990 con 0x7f73500d20d0 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f734d59a700 1 -- 172.21.15.89:0/115153622 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344011d20 con 0x7f73500d20d0 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f7354a02700 1 -- 172.21.15.89:0/115153622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73500d20d0 msgr2=0x7f73500b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f7354a02700 1 --2- 172.21.15.89:0/115153622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73500d20d0 0x7f73500b18c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7344009840 tx=0x7f7344009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f7354a02700 1 -- 172.21.15.89:0/115153622 shutdown_connections 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f7354a02700 1 --2- 172.21.15.89:0/115153622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735015e160 0x7f73500d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f7354a02700 1 --2- 172.21.15.89:0/115153622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73500d20d0 0x7f73500b18c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f7354a02700 1 -- 172.21.15.89:0/115153622 >> 172.21.15.89:0/115153622 conn(0x7f7350073130 msgr2=0x7f735009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f7354a02700 1 -- 172.21.15.89:0/115153622 shutdown_connections 2023-11-24T06:19:40.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.887+0000 7f7354a02700 1 -- 172.21.15.89:0/115153622 wait complete. 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.888+0000 7f7354a02700 1 Processor -- start 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f7354a02700 1 -- start start 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f7354a02700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73500b1e00 0x7f73500c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f7354a02700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73500d20d0 0x7f7350159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f7354a02700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f735015e160 0x7f735015a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f7354a02700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735015a9a0 con 0x7f735015e160 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f7354a02700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73500b0170 con 0x7f73500d20d0 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f7354a02700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735015f5a0 con 0x7f73500b1e00 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f734ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f735015e160 0x7f735015a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f734ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f735015e160 0x7f735015a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36784/0 (socket says 172.21.15.89:36784) 2023-11-24T06:19:40.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f734ed9d700 1 -- 172.21.15.89:0/4026979500 learned_addr learned my addr 172.21.15.89:0/4026979500 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f734e59c700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73500b1e00 0x7f73500c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.889+0000 7f734dd9b700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73500d20d0 0x7f7350159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f734ed9d700 1 -- 172.21.15.89:0/4026979500 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73500b1e00 msgr2=0x7f73500c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f734ed9d700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73500b1e00 0x7f73500c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f734ed9d700 1 -- 172.21.15.89:0/4026979500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73500d20d0 msgr2=0x7f7350159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f734ed9d700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73500d20d0 0x7f7350159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f734ed9d700 1 -- 172.21.15.89:0/4026979500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73440094a0 con 0x7f735015e160 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f734ed9d700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f735015e160 0x7f735015a460 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7348002230 tx=0x7f734800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f733effd700 1 -- 172.21.15.89:0/4026979500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734800aac0 con 0x7f735015e160 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f733effd700 1 -- 172.21.15.89:0/4026979500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734800ac40 con 0x7f735015e160 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735015f880 con 0x7f735015e160 2023-11-24T06:19:40.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f733effd700 1 -- 172.21.15.89:0/4026979500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73480070d0 con 0x7f735015e160 2023-11-24T06:19:40.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.890+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73500b4ae0 con 0x7f735015e160 2023-11-24T06:19:40.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.891+0000 7f733effd700 1 -- 172.21.15.89:0/4026979500 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7348007250 con 0x7f735015e160 2023-11-24T06:19:40.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.892+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7350068fb0 con 0x7f735015e160 2023-11-24T06:19:40.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.892+0000 7f733effd700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f733006aed0 0x7f733006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:40.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.892+0000 7f733effd700 1 -- 172.21.15.89:0/4026979500 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7348089010 con 0x7f735015e160 2023-11-24T06:19:40.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.892+0000 7f734e59c700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f733006aed0 0x7f733006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:40.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.892+0000 7f734e59c700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f733006aed0 0x7f733006d390 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f734000c940 tx=0x7f7340008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:40.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:40.896+0000 7f733effd700 1 -- 172.21.15.89:0/4026979500 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73480544a0 con 0x7f735015e160 2023-11-24T06:19:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.085+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f735009b120 con 0x7f735015e160 2023-11-24T06:19:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.085+0000 7f733effd700 1 -- 172.21.15.89:0/4026979500 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f73480180a0 con 0x7f735015e160 2023-11-24T06:19:41.087 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:19:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f733006aed0 msgr2=0x7f733006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f733006aed0 0x7f733006d390 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f734000c940 tx=0x7f7340008040 comp rx=0 tx=0).stop 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f735015e160 msgr2=0x7f735015a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f735015e160 0x7f735015a460 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7348002230 tx=0x7f734800cf50 comp rx=0 tx=0).stop 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 shutdown_connections 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f735015e160 0x7f735015a460 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73500b1e00 0x7f73500c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f733006aed0 0x7f733006d390 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 --2- 172.21.15.89:0/4026979500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73500d20d0 0x7f7350159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 >> 172.21.15.89:0/4026979500 conn(0x7f7350073130 msgr2=0x7f73500cc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 shutdown_connections 2023-11-24T06:19:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.087+0000 7f7354a02700 1 -- 172.21.15.89:0/4026979500 wait complete. 2023-11-24T06:19:41.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T06:19:41.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.219+0000 7f43b2dfa700 1 Processor -- start 2023-11-24T06:19:41.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.219+0000 7f43b2dfa700 1 -- start start 2023-11-24T06:19:41.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.220+0000 7f43b2dfa700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f43ac15bdd0 0x7f43ac1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.220+0000 7f43b2dfa700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac0b3370 0x7f43ac0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.220+0000 7f43b2dfa700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43ac0d1c70 0x7f43ac0d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.220+0000 7f43b2dfa700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ac0b2f50 con 0x7f43ac0d1c70 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.220+0000 7f43b2dfa700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ac0d2660 con 0x7f43ac15bdd0 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.220+0000 7f43b2dfa700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0b2030 con 0x7f43ac0b3370 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.220+0000 7f43b1397700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43ac0d1c70 0x7f43ac0d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.220+0000 7f43b1397700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43ac0d1c70 0x7f43ac0d2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36790/0 (socket says 172.21.15.89:36790) 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43b1397700 1 -- 172.21.15.89:0/156435461 learned_addr learned my addr 172.21.15.89:0/156435461 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:41.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43b0b96700 1 --2- 172.21.15.89:0/156435461 >> v2:172.21.15.123:3300/0 conn(0x7f43ac15bdd0 0x7f43ac1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43abfff700 1 --2- 172.21.15.89:0/156435461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac0b3370 0x7f43ac0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43b1397700 1 -- 172.21.15.89:0/156435461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac0b3370 msgr2=0x7f43ac0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43b1397700 1 --2- 172.21.15.89:0/156435461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac0b3370 0x7f43ac0b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43b1397700 1 -- 172.21.15.89:0/156435461 >> v2:172.21.15.123:3300/0 conn(0x7f43ac15bdd0 msgr2=0x7f43ac1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43b1397700 1 --2- 172.21.15.89:0/156435461 >> v2:172.21.15.123:3300/0 conn(0x7f43ac15bdd0 0x7f43ac1604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43b1397700 1 -- 172.21.15.89:0/156435461 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac0b22d0 con 0x7f43ac0d1c70 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43b1397700 1 --2- 172.21.15.89:0/156435461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0d2120 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f43ac06ec10 tx=0x7f43a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e780a9a9c28a57 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43ab7fe700 1 -- 172.21.15.89:0/156435461 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a400e070 con 0x7f43ac0d1c70 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43ab7fe700 1 -- 172.21.15.89:0/156435461 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a400b910 con 0x7f43ac0d1c70 2023-11-24T06:19:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.221+0000 7f43ab7fe700 1 -- 172.21.15.89:0/156435461 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a400bcc0 con 0x7f43ac0d1c70 2023-11-24T06:19:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 -- 172.21.15.89:0/156435461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0d1c70 msgr2=0x7f43ac0d2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/156435461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0d2120 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f43ac06ec10 tx=0x7f43a400b330 comp rx=0 tx=0).stop 2023-11-24T06:19:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 -- 172.21.15.89:0/156435461 shutdown_connections 2023-11-24T06:19:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/156435461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0d2120 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/156435461 >> v2:172.21.15.123:3300/0 conn(0x7f43ac15bdd0 0x7f43ac1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/156435461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac0b3370 0x7f43ac0b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 -- 172.21.15.89:0/156435461 >> 172.21.15.89:0/156435461 conn(0x7f43ac0731a0 msgr2=0x7f43ac09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 -- 172.21.15.89:0/156435461 shutdown_connections 2023-11-24T06:19:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.222+0000 7f43b2dfa700 1 -- 172.21.15.89:0/156435461 wait complete. 2023-11-24T06:19:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.223+0000 7f43b2dfa700 1 Processor -- start 2023-11-24T06:19:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.223+0000 7f43b2dfa700 1 -- start start 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b2dfa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0b3370 0x7f43ac0b7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b2dfa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b2dfa700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac15bdd0 0x7f43ac14d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b2dfa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac14d540 con 0x7f43ac0b3370 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b2dfa700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0b62d0 con 0x7f43ac0d1c70 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b2dfa700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0b65b0 con 0x7f43ac15bdd0 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0b3370 0x7f43ac0b7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0b3370 0x7f43ac0b7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36792/0 (socket says 172.21.15.89:36792) 2023-11-24T06:19:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 -- 172.21.15.89:0/335068084 learned_addr learned my addr 172.21.15.89:0/335068084 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43abfff700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b1397700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac15bdd0 0x7f43ac14d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 -- 172.21.15.89:0/335068084 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac15bdd0 msgr2=0x7f43ac14d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac15bdd0 0x7f43ac14d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 -- 172.21.15.89:0/335068084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43ac0d1c70 msgr2=0x7f43ac0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 -- 172.21.15.89:0/335068084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4009b30 con 0x7f43ac0b3370 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b0b96700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0b3370 0x7f43ac0b7370 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f43a000e9b0 tx=0x7f43a000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43a97fa700 1 -- 172.21.15.89:0/335068084 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0009710 con 0x7f43ac0b3370 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43a97fa700 1 -- 172.21.15.89:0/335068084 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a000c850 con 0x7f43ac0b3370 2023-11-24T06:19:41.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac0b6890 con 0x7f43ac0b3370 2023-11-24T06:19:41.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.224+0000 7f43a97fa700 1 -- 172.21.15.89:0/335068084 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a000cbc0 con 0x7f43ac0b3370 2023-11-24T06:19:41.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.225+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ac0b6e50 con 0x7f43ac0b3370 2023-11-24T06:19:41.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.226+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ac067f60 con 0x7f43ac0b3370 2023-11-24T06:19:41.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.226+0000 7f43a97fa700 1 -- 172.21.15.89:0/335068084 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f43a000cd20 con 0x7f43ac0b3370 2023-11-24T06:19:41.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.226+0000 7f43a97fa700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f439006aed0 0x7f439006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.226+0000 7f43a97fa700 1 -- 172.21.15.89:0/335068084 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f43a0089380 con 0x7f43ac0b3370 2023-11-24T06:19:41.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.226+0000 7f43abfff700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f439006aed0 0x7f439006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.227+0000 7f43abfff700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f439006aed0 0x7f439006d390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f43ac0c8160 tx=0x7f4398008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:41.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.231+0000 7f43a97fa700 1 -- 172.21.15.89:0/335068084 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43a0054780 con 0x7f43ac0b3370 2023-11-24T06:19:41.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.421+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f43ac0b7c60 con 0x7f43ac0b3370 2023-11-24T06:19:41.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.422+0000 7f43a97fa700 1 -- 172.21.15.89:0/335068084 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f43a0021060 con 0x7f43ac0b3370 2023-11-24T06:19:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:41.425 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-11-24T06:19:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2023-11-24T06:19:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2023-11-24T06:19:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:41.426 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-11-24T06:19:41.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f439006aed0 msgr2=0x7f439006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f439006aed0 0x7f439006d390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f43ac0c8160 tx=0x7f4398008040 comp rx=0 tx=0).stop 2023-11-24T06:19:41.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0b3370 msgr2=0x7f43ac0b7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0b3370 0x7f43ac0b7370 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f43a000e9b0 tx=0x7f43a000ed70 comp rx=0 tx=0).stop 2023-11-24T06:19:41.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 shutdown_connections 2023-11-24T06:19:41.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43ac15bdd0 0x7f43ac14d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43ac0b3370 0x7f43ac0b7370 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f439006aed0 0x7f439006d390 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 --2- 172.21.15.89:0/335068084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 >> 172.21.15.89:0/335068084 conn(0x7f43ac0731a0 msgr2=0x7f43ac0b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:41.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 shutdown_connections 2023-11-24T06:19:41.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.424+0000 7f43b2dfa700 1 -- 172.21.15.89:0/335068084 wait complete. 2023-11-24T06:19:41.438 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T06:19:41.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T06:19:41.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.560+0000 7f1b6c4fc700 1 Processor -- start 2023-11-24T06:19:41.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.560+0000 7f1b6c4fc700 1 -- start start 2023-11-24T06:19:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6c4fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b640b01e0 0x7f1b6406e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6c4fc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1b640746a0 0x7f1b640b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6c4fc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b640b05c0 con 0x7f1b640b59c0 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6c4fc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b641470a0 con 0x7f1b640b01e0 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6c4fc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b640b5e70 con 0x7f1b640746a0 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6a298700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b640b01e0 0x7f1b6406e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6a298700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b640b01e0 0x7f1b6406e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36800/0 (socket says 172.21.15.89:36800) 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6a298700 1 -- 172.21.15.89:0/3062546297 learned_addr learned my addr 172.21.15.89:0/3062546297 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6aa99700 1 --2- 172.21.15.89:0/3062546297 >> v2:172.21.15.123:3300/0 conn(0x7f1b640746a0 0x7f1b640b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6a298700 1 -- 172.21.15.89:0/3062546297 >> v2:172.21.15.123:3300/0 conn(0x7f1b640746a0 msgr2=0x7f1b640b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.561+0000 7f1b6a298700 1 --2- 172.21.15.89:0/3062546297 >> v2:172.21.15.123:3300/0 conn(0x7f1b640746a0 0x7f1b640b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6a298700 1 -- 172.21.15.89:0/3062546297 >> v1:172.21.15.123:6789/0 conn(0x7f1b640b59c0 legacy=0x7f1b64073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6a298700 1 -- 172.21.15.89:0/3062546297 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b640b4250 con 0x7f1b640b01e0 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6a298700 1 --2- 172.21.15.89:0/3062546297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640b01e0 0x7f1b6406e9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1b640d1970 tx=0x7f1b54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3546fcd639e79374 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b69296700 1 -- 172.21.15.89:0/3062546297 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54004230 con 0x7f1b640b01e0 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b69296700 1 -- 172.21.15.89:0/3062546297 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b54004390 con 0x7f1b640b01e0 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b69296700 1 -- 172.21.15.89:0/3062546297 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b540036a0 con 0x7f1b640b01e0 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/3062546297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640b01e0 msgr2=0x7f1b6406e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/3062546297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640b01e0 0x7f1b6406e9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1b640d1970 tx=0x7f1b54009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/3062546297 shutdown_connections 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/3062546297 >> v2:172.21.15.123:3300/0 conn(0x7f1b640746a0 0x7f1b640b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/3062546297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640b01e0 0x7f1b6406e9c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.562+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/3062546297 >> 172.21.15.89:0/3062546297 conn(0x7f1b640d0b40 msgr2=0x7f1b640d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.563+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/3062546297 shutdown_connections 2023-11-24T06:19:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.563+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/3062546297 wait complete. 2023-11-24T06:19:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6c4fc700 1 Processor -- start 2023-11-24T06:19:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6c4fc700 1 -- start start 2023-11-24T06:19:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6c4fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640746a0 0x7f1b6415f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6c4fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b640b01e0 0x7f1b6415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6c4fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b640b59c0 0x7f1b6415c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6c4fc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6415f970 con 0x7f1b640746a0 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6c4fc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6415c370 con 0x7f1b640b59c0 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6c4fc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b64161c10 con 0x7f1b640b01e0 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6a298700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640746a0 0x7f1b6415f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.564+0000 7f1b6a298700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640746a0 0x7f1b6415f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36806/0 (socket says 172.21.15.89:36806) 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6a298700 1 -- 172.21.15.89:0/794972784 learned_addr learned my addr 172.21.15.89:0/794972784 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6aa99700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b640b59c0 0x7f1b6415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b69a97700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b640b01e0 0x7f1b6415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6a298700 1 -- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b640b01e0 msgr2=0x7f1b6415be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6a298700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b640b01e0 0x7f1b6415be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6a298700 1 -- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b640b59c0 msgr2=0x7f1b6415c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6a298700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b640b59c0 0x7f1b6415c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6a298700 1 -- 172.21.15.89:0/794972784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b54009550 con 0x7f1b640746a0 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b69a97700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b640b01e0 0x7f1b6415be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6a298700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640746a0 0x7f1b6415f430 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009870 tx=0x7f1b54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b5affd700 1 -- 172.21.15.89:0/794972784 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54003e30 con 0x7f1b640746a0 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b5affd700 1 -- 172.21.15.89:0/794972784 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5401f070 con 0x7f1b640746a0 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6415e670 con 0x7f1b640746a0 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b5affd700 1 -- 172.21.15.89:0/794972784 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b54011870 con 0x7f1b640746a0 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.565+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6415ec30 con 0x7f1b640746a0 2023-11-24T06:19:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.566+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b64067f60 con 0x7f1b640746a0 2023-11-24T06:19:41.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.567+0000 7f1b5affd700 1 -- 172.21.15.89:0/794972784 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1b54021030 con 0x7f1b640746a0 2023-11-24T06:19:41.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.567+0000 7f1b5affd700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b4806aed0 0x7f1b4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:41.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.567+0000 7f1b5affd700 1 -- 172.21.15.89:0/794972784 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1b5408a8b0 con 0x7f1b640746a0 2023-11-24T06:19:41.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.567+0000 7f1b69a97700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b4806aed0 0x7f1b4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:41.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.568+0000 7f1b69a97700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b4806aed0 0x7f1b4806d390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1b6406c480 tx=0x7f1b60009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:41.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.571+0000 7f1b5affd700 1 -- 172.21.15.89:0/794972784 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b54055d40 con 0x7f1b640746a0 2023-11-24T06:19:41.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:41.765+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1b6415f280 con 0x7f1b640746a0 2023-11-24T06:19:42.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.581+0000 7f1b5affd700 1 -- 172.21.15.89:0/794972784 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1b54059360 con 0x7f1b640746a0 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b4806aed0 msgr2=0x7f1b4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b4806aed0 0x7f1b4806d390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1b6406c480 tx=0x7f1b60009040 comp rx=0 tx=0).stop 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640746a0 msgr2=0x7f1b6415f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640746a0 0x7f1b6415f430 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009870 tx=0x7f1b54004750 comp rx=0 tx=0).stop 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 shutdown_connections 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b4806aed0 0x7f1b4806d390 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b640b59c0 0x7f1b6415c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640746a0 0x7f1b6415f430 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 --2- 172.21.15.89:0/794972784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b640b01e0 0x7f1b6415be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.584+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 >> 172.21.15.89:0/794972784 conn(0x7f1b640d0b40 msgr2=0x7f1b64061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:42.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.585+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 shutdown_connections 2023-11-24T06:19:42.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.585+0000 7f1b6c4fc700 1 -- 172.21.15.89:0/794972784 wait complete. 2023-11-24T06:19:42.591 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2023-11-24T06:19:42.603 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T06:19:42.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.733+0000 7f474d856700 1 Processor -- start 2023-11-24T06:19:42.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.734+0000 7f474d856700 1 -- start start 2023-11-24T06:19:42.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.734+0000 7f474d856700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480d05b0 0x7f47480b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:42.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.734+0000 7f474d856700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47480b2a60 con 0x7f474806e930 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.734+0000 7f474d856700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474815e970 con 0x7f474815ac40 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.734+0000 7f474d856700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474815ef30 con 0x7f47480d05b0 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f47477fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f474806e930 0x7f47480ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40322/0 (socket says 172.21.15.89:40322) 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f47477fe700 1 -- 172.21.15.89:0/3046443956 learned_addr learned my addr 172.21.15.89:0/3046443956 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f47467fc700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480d05b0 0x7f47480b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911680817 0 0) 0x7f47480b2a60 con 0x7f474806e930 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47300035b0 con 0x7f474806e930 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065691598 0 0) 0x7f474815e970 con 0x7f474815ac40 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47480b2a60 con 0x7f474815ac40 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635616664 0 0) 0x7f47300035b0 con 0x7f474806e930 2023-11-24T06:19:42.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480d05b0 msgr2=0x7f47480b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480d05b0 0x7f47480b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 >> v1:172.21.15.123:6789/0 conn(0x7f474815ac40 legacy=0x7f47480cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47480b2cb0 con 0x7f474806e930 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f47400019f0 con 0x7f474806e930 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 >> v1:172.21.15.89:6789/0 conn(0x7f474806e930 legacy=0x7f47480ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4730003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480d05b0 0x7f4730004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4730004d90 0x7f4730005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474815ef30 con 0x7f4730003bc0 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473000bb10 con 0x7f4730004d90 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473000bcd0 con 0x7f47480d05b0 2023-11-24T06:19:42.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f4746ffd700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4730003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f47467fc700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480d05b0 0x7f4730004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.735+0000 7f47477fe700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4730004d90 0x7f4730005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4746ffd700 1 -- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480d05b0 msgr2=0x7f4730004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4746ffd700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480d05b0 0x7f4730004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4746ffd700 1 -- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4730004d90 msgr2=0x7f4730005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4746ffd700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4730004d90 0x7f4730005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4746ffd700 1 -- 172.21.15.89:0/3046443956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47300035b0 con 0x7f4730003bc0 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4746ffd700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4730003fc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f473c001940 tx=0x7f4730005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e4469d24201f15 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c001430 con 0x7f4730003bc0 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c001d70 con 0x7f4730003bc0 2023-11-24T06:19:42.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f4745ffb700 1 -- 172.21.15.89:0/3046443956 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c005510 con 0x7f4730003bc0 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f474d856700 1 -- 172.21.15.89:0/3046443956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 msgr2=0x7f4730003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f474d856700 1 --2- 172.21.15.89:0/3046443956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4730003fc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f473c001940 tx=0x7f4730005880 comp rx=0 tx=0).stop 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f47477fe700 1 -- 172.21.15.89:0/3046443956 reap_dead start 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f474d856700 1 -- 172.21.15.89:0/3046443956 shutdown_connections 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f474d856700 1 -- 172.21.15.89:0/3046443956 >> 172.21.15.89:0/3046443956 conn(0x7f474806b410 msgr2=0x7f474806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f474d856700 1 -- 172.21.15.89:0/3046443956 shutdown_connections 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.736+0000 7f474d856700 1 -- 172.21.15.89:0/3046443956 wait complete. 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.738+0000 7f474d856700 1 Processor -- start 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.738+0000 7f474d856700 1 -- start start 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.738+0000 7f474d856700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4748143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.738+0000 7f474d856700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480c9220 0x7f47481f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f474d856700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4748159650 0x7f4748159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f474d856700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474815a040 con 0x7f4730003bc0 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f474d856700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47480c9a50 con 0x7f4748159650 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f474d856700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47480c9d30 con 0x7f47480c9220 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4748143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4748143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36826/0 (socket says 172.21.15.89:36826) 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 -- 172.21.15.89:0/3507387069 learned_addr learned my addr 172.21.15.89:0/3507387069 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f47467fc700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480c9220 0x7f47481f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f47477fe700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4748159650 0x7f4748159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 -- 172.21.15.89:0/3507387069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480c9220 msgr2=0x7f47481f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480c9220 0x7f47481f3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 -- 172.21.15.89:0/3507387069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4748159650 msgr2=0x7f4748159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:42.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4748159650 0x7f4748159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:42.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 -- 172.21.15.89:0/3507387069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47480ca010 con 0x7f4730003bc0 2023-11-24T06:19:42.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f4746ffd700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4748143c30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f473c00c3c0 tx=0x7f473c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:42.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f472ffff700 1 -- 172.21.15.89:0/3507387069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c006c70 con 0x7f4730003bc0 2023-11-24T06:19:42.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f472ffff700 1 -- 172.21.15.89:0/3507387069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c006dd0 con 0x7f4730003bc0 2023-11-24T06:19:42.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47480ca2f0 con 0x7f4730003bc0 2023-11-24T06:19:42.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f472ffff700 1 -- 172.21.15.89:0/3507387069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c007280 con 0x7f4730003bc0 2023-11-24T06:19:42.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.739+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47481f3dd0 con 0x7f4730003bc0 2023-11-24T06:19:42.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.740+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4748068480 con 0x7f4730003bc0 2023-11-24T06:19:42.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.741+0000 7f472ffff700 1 -- 172.21.15.89:0/3507387069 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f473c0073e0 con 0x7f4730003bc0 2023-11-24T06:19:42.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.741+0000 7f472ffff700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f473406aed0 0x7f473406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:42.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.741+0000 7f472ffff700 1 -- 172.21.15.89:0/3507387069 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f473c08a220 con 0x7f4730003bc0 2023-11-24T06:19:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.741+0000 7f47467fc700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f473406aed0 0x7f473406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.741+0000 7f47467fc700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f473406aed0 0x7f473406d390 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4748147010 tx=0x7f4738008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.746+0000 7f472ffff700 1 -- 172.21.15.89:0/3507387069 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f473c055540 con 0x7f4730003bc0 2023-11-24T06:19:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:42.936+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4748068fb0 con 0x7f4730003bc0 2023-11-24T06:19:43.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.584+0000 7f472ffff700 1 -- 172.21.15.89:0/3507387069 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f473c058b60 con 0x7f4730003bc0 2023-11-24T06:19:43.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.586+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f473406aed0 msgr2=0x7f473406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:43.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.586+0000 7f474d856700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f473406aed0 0x7f473406d390 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4748147010 tx=0x7f4738008040 comp rx=0 tx=0).stop 2023-11-24T06:19:43.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.586+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 msgr2=0x7f4748143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:43.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.587+0000 7f474d856700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4748143c30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f473c00c3c0 tx=0x7f473c0012d0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.589+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 shutdown_connections 2023-11-24T06:19:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.589+0000 7f474d856700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f473406aed0 0x7f473406d390 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.589+0000 7f474d856700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4748159650 0x7f4748159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.589+0000 7f474d856700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4730003bc0 0x7f4748143c30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.589+0000 7f474d856700 1 --2- 172.21.15.89:0/3507387069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47480c9220 0x7f47481f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.589+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 >> 172.21.15.89:0/3507387069 conn(0x7f474806b410 msgr2=0x7f47480b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:43.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.591+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 shutdown_connections 2023-11-24T06:19:43.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.591+0000 7f474d856700 1 -- 172.21.15.89:0/3507387069 wait complete. 2023-11-24T06:19:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2023-11-24T06:19:43.619 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T06:19:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.744+0000 7fcadcab3700 1 Processor -- start 2023-11-24T06:19:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.744+0000 7fcadcab3700 1 -- start start 2023-11-24T06:19:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.745+0000 7fcadcab3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcad80746a0 0x7fcad80b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.745+0000 7fcadcab3700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad80b3c60 con 0x7fcad80afc90 2023-11-24T06:19:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.745+0000 7fcadcab3700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad80b58c0 con 0x7fcad815e160 2023-11-24T06:19:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.745+0000 7fcadcab3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80b4500 con 0x7fcad80746a0 2023-11-24T06:19:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.745+0000 7fcad6d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcad80afc90 0x7fcad806d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40342/0 (socket says 172.21.15.89:40342) 2023-11-24T06:19:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.745+0000 7fcad6d9d700 1 -- 172.21.15.89:0/728692017 learned_addr learned my addr 172.21.15.89:0/728692017 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385240459 0 0) 0x7fcad80b3c60 con 0x7fcad80afc90 2023-11-24T06:19:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac00035b0 con 0x7fcad80afc90 2023-11-24T06:19:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad5d9b700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcad80746a0 0x7fcad80b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4057869888 0 0) 0x7fcac00035b0 con 0x7fcad80afc90 2023-11-24T06:19:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcad80746a0 msgr2=0x7fcad80b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcad80746a0 0x7fcad80b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 >> v1:172.21.15.123:6789/0 conn(0x7fcad815e160 legacy=0x7fcad8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad80b0140 con 0x7fcad80afc90 2023-11-24T06:19:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fcad00019f0 con 0x7fcad80afc90 2023-11-24T06:19:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 >> v1:172.21.15.89:6789/0 conn(0x7fcad80afc90 legacy=0x7fcad806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcac0003a30 0x7fcac0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcad80746a0 0x7fcac0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac0004d90 0x7fcac0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80b4500 con 0x7fcac0003a30 2023-11-24T06:19:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac000bba0 con 0x7fcac0004d90 2023-11-24T06:19:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac000bd30 con 0x7fcad80746a0 2023-11-24T06:19:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad659c700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcac0003a30 0x7fcac0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad5d9b700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcad80746a0 0x7fcac0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad6d9d700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac0004d90 0x7fcac0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad659c700 1 -- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcad80746a0 msgr2=0x7fcac0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad659c700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcad80746a0 0x7fcac0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad659c700 1 -- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac0004d90 msgr2=0x7fcac0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad659c700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac0004d90 0x7fcac0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.746+0000 7fcad659c700 1 -- 172.21.15.89:0/728692017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac00035b0 con 0x7fcac0003a30 2023-11-24T06:19:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcad659c700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcac0003a30 0x7fcac0003e50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcacc0017f0 tx=0x7fcac0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1641f6fcb075349e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0052d0 con 0x7fcac0003a30 2023-11-24T06:19:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcacc005430 con 0x7fcac0003a30 2023-11-24T06:19:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcad559a700 1 -- 172.21.15.89:0/728692017 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0038c0 con 0x7fcac0003a30 2023-11-24T06:19:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcadcab3700 1 -- 172.21.15.89:0/728692017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcac0003a30 msgr2=0x7fcac0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:43.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcadcab3700 1 --2- 172.21.15.89:0/728692017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcac0003a30 0x7fcac0003e50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcacc0017f0 tx=0x7fcac0005880 comp rx=0 tx=0).stop 2023-11-24T06:19:43.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcad6d9d700 1 -- 172.21.15.89:0/728692017 reap_dead start 2023-11-24T06:19:43.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcadcab3700 1 -- 172.21.15.89:0/728692017 shutdown_connections 2023-11-24T06:19:43.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcadcab3700 1 -- 172.21.15.89:0/728692017 >> 172.21.15.89:0/728692017 conn(0x7fcad80d0bf0 msgr2=0x7fcad80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:43.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcadcab3700 1 -- 172.21.15.89:0/728692017 shutdown_connections 2023-11-24T06:19:43.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.747+0000 7fcadcab3700 1 -- 172.21.15.89:0/728692017 wait complete. 2023-11-24T06:19:43.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.749+0000 7fcadcab3700 1 Processor -- start 2023-11-24T06:19:43.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.749+0000 7fcadcab3700 1 -- start start 2023-11-24T06:19:43.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.749+0000 7fcadcab3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac0003a30 0x7fcad814e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:43.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.749+0000 7fcadcab3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad81607e0 0x7fcad8160c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:43.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.749+0000 7fcadcab3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcad815c270 0x7fcad815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:43.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcadcab3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8161590 con 0x7fcad815c270 2023-11-24T06:19:43.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcadcab3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad815cc30 con 0x7fcad81607e0 2023-11-24T06:19:43.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcadcab3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad814ec20 con 0x7fcac0003a30 2023-11-24T06:19:43.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcad815c270 0x7fcad815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:43.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcad815c270 0x7fcad815c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36848/0 (socket says 172.21.15.89:36848) 2023-11-24T06:19:43.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 -- 172.21.15.89:0/4288517060 learned_addr learned my addr 172.21.15.89:0/4288517060 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:43.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad659c700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac0003a30 0x7fcad814e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:43.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad5d9b700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad81607e0 0x7fcad8160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:43.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 -- 172.21.15.89:0/4288517060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac0003a30 msgr2=0x7fcad814e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:43.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac0003a30 0x7fcad814e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 -- 172.21.15.89:0/4288517060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad81607e0 msgr2=0x7fcad8160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:43.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad81607e0 0x7fcad8160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:43.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 -- 172.21.15.89:0/4288517060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad81612b0 con 0x7fcad815c270 2023-11-24T06:19:43.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcad6d9d700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcad815c270 0x7fcad815c6f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcad8162db0 tx=0x7fcad000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:43.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcac6ffd700 1 -- 172.21.15.89:0/4288517060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad000ad40 con 0x7fcad815c270 2023-11-24T06:19:43.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcac6ffd700 1 -- 172.21.15.89:0/4288517060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad00133f0 con 0x7fcad815c270 2023-11-24T06:19:43.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcac6ffd700 1 -- 172.21.15.89:0/4288517060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad000aea0 con 0x7fcad815c270 2023-11-24T06:19:43.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad80c5d70 con 0x7fcad815c270 2023-11-24T06:19:43.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.750+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad80c6330 con 0x7fcad815c270 2023-11-24T06:19:43.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.751+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad80c5bd0 con 0x7fcad815c270 2023-11-24T06:19:43.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.752+0000 7fcac6ffd700 1 -- 172.21.15.89:0/4288517060 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcad00137a0 con 0x7fcad815c270 2023-11-24T06:19:43.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.752+0000 7fcac6ffd700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcab806aed0 0x7fcab806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:43.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.753+0000 7fcad659c700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcab806aed0 0x7fcab806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:43.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.753+0000 7fcac6ffd700 1 -- 172.21.15.89:0/4288517060 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fcad0089db0 con 0x7fcad815c270 2023-11-24T06:19:43.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.753+0000 7fcad659c700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcab806aed0 0x7fcab806d390 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcacc000c00 tx=0x7fcacc009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:43.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.757+0000 7fcac6ffd700 1 -- 172.21.15.89:0/4288517060 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcad80c5bd0 con 0x7fcad815c270 2023-11-24T06:19:43.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:43.988+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcad80c5bd0 con 0x7fcad815c270 2023-11-24T06:19:44.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.980+0000 7fcac6ffd700 1 -- 172.21.15.89:0/4288517060 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fcad80c5bd0 con 0x7fcad815c270 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcab806aed0 msgr2=0x7fcab806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcab806aed0 0x7fcab806d390 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcacc000c00 tx=0x7fcacc009030 comp rx=0 tx=0).stop 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcad815c270 msgr2=0x7fcad815c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcad815c270 0x7fcad815c6f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcad8162db0 tx=0x7fcad000a5f0 comp rx=0 tx=0).stop 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 shutdown_connections 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcad815c270 0x7fcad815c6f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac0003a30 0x7fcad814e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcab806aed0 0x7fcab806d390 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 --2- 172.21.15.89:0/4288517060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad81607e0 0x7fcad8160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:44.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 >> 172.21.15.89:0/4288517060 conn(0x7fcad80d0bf0 msgr2=0x7fcad806d930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:44.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 shutdown_connections 2023-11-24T06:19:44.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:44.984+0000 7fcadcab3700 1 -- 172.21.15.89:0/4288517060 wait complete. 2023-11-24T06:19:44.988 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 4 and data pool 5 2023-11-24T06:19:45.005 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d694e8700 1 Processor -- start 2023-11-24T06:19:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d694e8700 1 -- start start 2023-11-24T06:19:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d694e8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158670 0x7f2d640622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d694e8700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d64158a50 0x7f2d640c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d694e8700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2d64162a30 0x7f2d640c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d694e8700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d640cfff0 con 0x7f2d64158a50 2023-11-24T06:19:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d694e8700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d640cfd10 con 0x7f2d64162a30 2023-11-24T06:19:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d694e8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d64161f50 con 0x7f2d64158670 2023-11-24T06:19:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.128+0000 7f2d627fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d64158a50 0x7f2d640c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d627fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d64158a50 0x7f2d640c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36854/0 (socket says 172.21.15.89:36854) 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d627fc700 1 -- 172.21.15.89:0/699486022 learned_addr learned my addr 172.21.15.89:0/699486022 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d62ffd700 1 --2- 172.21.15.89:0/699486022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158670 0x7f2d640622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d637fe700 1 --2- 172.21.15.89:0/699486022 >> v2:172.21.15.123:3300/0 conn(0x7f2d64162a30 0x7f2d640c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d627fc700 1 -- 172.21.15.89:0/699486022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158670 msgr2=0x7f2d640622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d627fc700 1 --2- 172.21.15.89:0/699486022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158670 0x7f2d640622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d627fc700 1 -- 172.21.15.89:0/699486022 >> v2:172.21.15.123:3300/0 conn(0x7f2d64162a30 msgr2=0x7f2d640c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d627fc700 1 --2- 172.21.15.89:0/699486022 >> v2:172.21.15.123:3300/0 conn(0x7f2d64162a30 0x7f2d640c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d627fc700 1 -- 172.21.15.89:0/699486022 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d64162230 con 0x7f2d64158a50 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d637fe700 1 --2- 172.21.15.89:0/699486022 >> v2:172.21.15.123:3300/0 conn(0x7f2d64162a30 0x7f2d640c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d627fc700 1 --2- 172.21.15.89:0/699486022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64158a50 0x7f2d640c98b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2d640b0940 tx=0x7f2d58009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2053ff5753f1f89b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d61ffb700 1 -- 172.21.15.89:0/699486022 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5800b870 con 0x7f2d64158a50 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d61ffb700 1 -- 172.21.15.89:0/699486022 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5800b9d0 con 0x7f2d64158a50 2023-11-24T06:19:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.129+0000 7f2d61ffb700 1 -- 172.21.15.89:0/699486022 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5800bd60 con 0x7f2d64158a50 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 -- 172.21.15.89:0/699486022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64158a50 msgr2=0x7f2d640c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 --2- 172.21.15.89:0/699486022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64158a50 0x7f2d640c98b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2d640b0940 tx=0x7f2d58009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 -- 172.21.15.89:0/699486022 shutdown_connections 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 --2- 172.21.15.89:0/699486022 >> v2:172.21.15.123:3300/0 conn(0x7f2d64162a30 0x7f2d640c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 --2- 172.21.15.89:0/699486022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158670 0x7f2d640622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 --2- 172.21.15.89:0/699486022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64158a50 0x7f2d640c98b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 -- 172.21.15.89:0/699486022 >> 172.21.15.89:0/699486022 conn(0x7f2d640b57b0 msgr2=0x7f2d640b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 -- 172.21.15.89:0/699486022 shutdown_connections 2023-11-24T06:19:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.130+0000 7f2d694e8700 1 -- 172.21.15.89:0/699486022 wait complete. 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.131+0000 7f2d694e8700 1 Processor -- start 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.131+0000 7f2d694e8700 1 -- start start 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d694e8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d64158670 0x7f2d641f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d694e8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158a50 0x7f2d640ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d694e8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64162a30 0x7f2d640ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d694e8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d640b3f50 con 0x7f2d64162a30 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d694e8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d640b3080 con 0x7f2d64158670 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d694e8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d640b32d0 con 0x7f2d64158a50 2023-11-24T06:19:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64162a30 0x7f2d640ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64162a30 0x7f2d640ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36862/0 (socket says 172.21.15.89:36862) 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 -- 172.21.15.89:0/3152890146 learned_addr learned my addr 172.21.15.89:0/3152890146 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d62ffd700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d64158670 0x7f2d641f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d627fc700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158a50 0x7f2d640ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 -- 172.21.15.89:0/3152890146 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158a50 msgr2=0x7f2d640ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158a50 0x7f2d640ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 -- 172.21.15.89:0/3152890146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d64158670 msgr2=0x7f2d641f34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d64158670 0x7f2d641f34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 -- 172.21.15.89:0/3152890146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d58009550 con 0x7f2d64162a30 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.132+0000 7f2d637fe700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64162a30 0x7f2d640ce6e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c00cef0 tx=0x7f2d5c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.133+0000 7f2d4b7fe700 1 -- 172.21.15.89:0/3152890146 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c00ae50 con 0x7f2d64162a30 2023-11-24T06:19:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.133+0000 7f2d4b7fe700 1 -- 172.21.15.89:0/3152890146 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5c015440 con 0x7f2d64162a30 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.133+0000 7f2d4b7fe700 1 -- 172.21.15.89:0/3152890146 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0157f0 con 0x7f2d64162a30 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.133+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d640b35b0 con 0x7f2d64162a30 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.133+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d640b3b70 con 0x7f2d64162a30 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.133+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6415fe20 con 0x7f2d64162a30 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.134+0000 7f2d4b7fe700 1 -- 172.21.15.89:0/3152890146 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2d5c00d430 con 0x7f2d64162a30 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.134+0000 7f2d4b7fe700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2d5006aed0 0x7f2d5006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.135+0000 7f2d62ffd700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2d5006aed0 0x7f2d5006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.135+0000 7f2d4b7fe700 1 -- 172.21.15.89:0/3152890146 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d5c089bb0 con 0x7f2d64162a30 2023-11-24T06:19:45.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.135+0000 7f2d62ffd700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2d5006aed0 0x7f2d5006d390 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400ba10 tx=0x7f2d54008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.139+0000 7f2d4b7fe700 1 -- 172.21.15.89:0/3152890146 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d6415fe20 con 0x7f2d64162a30 2023-11-24T06:19:45.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.331+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d6415fe20 con 0x7f2d64162a30 2023-11-24T06:19:45.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.332+0000 7f2d4b7fe700 1 -- 172.21.15.89:0/3152890146 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16546 (secure 0 0 0) 0x7f2d6415fe20 con 0x7f2d64162a30 2023-11-24T06:19:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:45.335 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:19:43.989535+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:45.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2d5006aed0 msgr2=0x7f2d5006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2d5006aed0 0x7f2d5006d390 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400ba10 tx=0x7f2d54008040 comp rx=0 tx=0).stop 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64162a30 msgr2=0x7f2d640ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64162a30 0x7f2d640ce6e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c00cef0 tx=0x7f2d5c00a340 comp rx=0 tx=0).stop 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 shutdown_connections 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d64162a30 0x7f2d640ce6e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2d5006aed0 0x7f2d5006d390 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d64158670 0x7f2d641f34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 --2- 172.21.15.89:0/3152890146 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d64158a50 0x7f2d640ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 >> 172.21.15.89:0/3152890146 conn(0x7f2d640b57b0 msgr2=0x7f2d640726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 shutdown_connections 2023-11-24T06:19:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.334+0000 7f2d694e8700 1 -- 172.21.15.89:0/3152890146 wait complete. 2023-11-24T06:19:45.350 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:45.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.466+0000 7f9cdacc3700 1 Processor -- start 2023-11-24T06:19:45.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.466+0000 7f9cdacc3700 1 -- start start 2023-11-24T06:19:45.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cdacc3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415ae20 0x7f9cd40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cdacc3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9cd40b1a70 0x7f9cd4164b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cdacc3700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd40cfd40 con 0x7f9cd415b200 2023-11-24T06:19:45.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cdacc3700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd40d0020 con 0x7f9cd40b1a70 2023-11-24T06:19:45.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cdacc3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40affa0 con 0x7f9cd415ae20 2023-11-24T06:19:45.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cd9260700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9cd40b1a70 0x7f9cd4164b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cd3fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9cd415b200 0x7f9cd40b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40370/0 (socket says 172.21.15.89:40370) 2023-11-24T06:19:45.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cd3fff700 1 -- 172.21.15.89:0/2028503951 learned_addr learned my addr 172.21.15.89:0/2028503951 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:45.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cd8a5f700 1 --2- 172.21.15.89:0/2028503951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415ae20 0x7f9cd40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cd9260700 1 -- 172.21.15.89:0/2028503951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415ae20 msgr2=0x7f9cd40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:45.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cd9260700 1 --2- 172.21.15.89:0/2028503951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415ae20 0x7f9cd40b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cd9260700 1 -- 172.21.15.89:0/2028503951 >> v1:172.21.15.89:6789/0 conn(0x7f9cd415b200 legacy=0x7f9cd40b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cd9260700 1 -- 172.21.15.89:0/2028503951 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd40b0280 con 0x7f9cd40b1a70 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.467+0000 7f9cd37fe700 1 -- 172.21.15.89:0/2028503951 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494399903 0 0) 0x7f9cd40cfd40 con 0x7f9cd415b200 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cd9260700 1 --2- 172.21.15.89:0/2028503951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4164b50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009d90 tx=0x7f9ccc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6857531cd8c74f02 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cd37fe700 1 -- 172.21.15.89:0/2028503951 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00e070 con 0x7f9cd40b1a70 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cd37fe700 1 -- 172.21.15.89:0/2028503951 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc00bbd0 con 0x7f9cd40b1a70 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cd37fe700 1 -- 172.21.15.89:0/2028503951 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc012430 con 0x7f9cd40b1a70 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cdacc3700 1 -- 172.21.15.89:0/2028503951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 msgr2=0x7f9cd4164b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/2028503951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4164b50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009d90 tx=0x7f9ccc00b330 comp rx=0 tx=0).stop 2023-11-24T06:19:45.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cdacc3700 1 -- 172.21.15.89:0/2028503951 shutdown_connections 2023-11-24T06:19:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/2028503951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4164b50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/2028503951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415ae20 0x7f9cd40b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cdacc3700 1 -- 172.21.15.89:0/2028503951 >> 172.21.15.89:0/2028503951 conn(0x7f9cd40b57b0 msgr2=0x7f9cd40b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cdacc3700 1 -- 172.21.15.89:0/2028503951 shutdown_connections 2023-11-24T06:19:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.468+0000 7f9cdacc3700 1 -- 172.21.15.89:0/2028503951 wait complete. 2023-11-24T06:19:45.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.469+0000 7f9cdacc3700 1 Processor -- start 2023-11-24T06:19:45.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cdacc3700 1 -- start start 2023-11-24T06:19:45.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cdacc3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cdacc3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cd415ae20 0x7f9cd414e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cdacc3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415b200 0x7f9cd4155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cdacc3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd415d450 con 0x7f9cd40b1a70 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cdacc3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41578c0 con 0x7f9cd415ae20 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cdacc3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd415c7d0 con 0x7f9cd415b200 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cd8a5f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.470+0000 7f9cd8a5f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36870/0 (socket says 172.21.15.89:36870) 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd8a5f700 1 -- 172.21.15.89:0/1912932013 learned_addr learned my addr 172.21.15.89:0/1912932013 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd9260700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415b200 0x7f9cd4155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd3fff700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cd415ae20 0x7f9cd414e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd8a5f700 1 -- 172.21.15.89:0/1912932013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415b200 msgr2=0x7f9cd4155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd8a5f700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415b200 0x7f9cd4155260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd8a5f700 1 -- 172.21.15.89:0/1912932013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cd415ae20 msgr2=0x7f9cd414e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd8a5f700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cd415ae20 0x7f9cd414e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd8a5f700 1 -- 172.21.15.89:0/1912932013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc009a70 con 0x7f9cd40b1a70 2023-11-24T06:19:45.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd8a5f700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4157380 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800e810 tx=0x7f9cc800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd17fa700 1 -- 172.21.15.89:0/1912932013 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8013610 con 0x7f9cd40b1a70 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd17fa700 1 -- 172.21.15.89:0/1912932013 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc8013770 con 0x7f9cd40b1a70 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd415cab0 con 0x7f9cd40b1a70 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cd17fa700 1 -- 172.21.15.89:0/1912932013 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8013b20 con 0x7f9cd40b1a70 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.471+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd415d070 con 0x7f9cd40b1a70 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.472+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd4067f60 con 0x7f9cd40b1a70 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.472+0000 7f9cd17fa700 1 -- 172.21.15.89:0/1912932013 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9cc8013c80 con 0x7f9cd40b1a70 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.473+0000 7f9cd17fa700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cb806aed0 0x7f9cb806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.473+0000 7f9cd17fa700 1 -- 172.21.15.89:0/1912932013 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cc8089280 con 0x7f9cd40b1a70 2023-11-24T06:19:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.473+0000 7f9cd3fff700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cb806aed0 0x7f9cb806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.473+0000 7f9cd3fff700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cb806aed0 0x7f9cb806d390 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9cd40cae20 tx=0x7f9cc0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.477+0000 7f9cd17fa700 1 -- 172.21.15.89:0/1912932013 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cc8054340 con 0x7f9cd40b1a70 2023-11-24T06:19:45.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.671+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cd40cde80 con 0x7f9cd40b1a70 2023-11-24T06:19:45.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.672+0000 7f9cd17fa700 1 -- 172.21.15.89:0/1912932013 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f9cc8057960 con 0x7f9cd40b1a70 2023-11-24T06:19:45.676 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:45.676 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":34,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:19:45.595335+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:45.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cb806aed0 msgr2=0x7f9cb806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cb806aed0 0x7f9cb806d390 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9cd40cae20 tx=0x7f9cc0009040 comp rx=0 tx=0).stop 2023-11-24T06:19:45.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 msgr2=0x7f9cd4157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4157380 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800e810 tx=0x7f9cc800ebd0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 shutdown_connections 2023-11-24T06:19:45.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cd415b200 0x7f9cd4155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cd40b1a70 0x7f9cd4157380 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cb806aed0 0x7f9cb806d390 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 --2- 172.21.15.89:0/1912932013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cd415ae20 0x7f9cd414e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 >> 172.21.15.89:0/1912932013 conn(0x7f9cd40b57b0 msgr2=0x7f9cd4061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:45.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 shutdown_connections 2023-11-24T06:19:45.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.675+0000 7f9cdacc3700 1 -- 172.21.15.89:0/1912932013 wait complete. 2023-11-24T06:19:45.690 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T06:19:45.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.811+0000 7f93b72d4700 1 Processor -- start 2023-11-24T06:19:45.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.811+0000 7f93b72d4700 1 -- start start 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b72d4700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93b015e160 0x7f93b00b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b72d4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b00afc90 0x7f93b0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b72d4700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b00b3c60 con 0x7f93b00744d0 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b72d4700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b00b4500 con 0x7f93b015e160 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b72d4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b0074960 con 0x7f93b00afc90 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b5070700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93b015e160 0x7f93b00b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b5871700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f93b00744d0 0x7f93b006d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40384/0 (socket says 172.21.15.89:40384) 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b5871700 1 -- 172.21.15.89:0/151624821 learned_addr learned my addr 172.21.15.89:0/151624821 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b5070700 1 -- 172.21.15.89:0/151624821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b00afc90 msgr2=0x7f93b0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b5070700 1 --2- 172.21.15.89:0/151624821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b00afc90 0x7f93b0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b5070700 1 -- 172.21.15.89:0/151624821 >> v1:172.21.15.89:6789/0 conn(0x7f93b00744d0 legacy=0x7f93b006d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b5070700 1 -- 172.21.15.89:0/151624821 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00b58c0 con 0x7f93b015e160 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93a7fff700 1 -- 172.21.15.89:0/151624821 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635008552 0 0) 0x7f93b00b3c60 con 0x7f93b00744d0 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93b5070700 1 --2- 172.21.15.89:0/151624821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b015e160 0x7f93b00b5da0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009840 tx=0x7f93ac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801fe92ddd8dfe9e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93a7fff700 1 -- 172.21.15.89:0/151624821 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac011830 con 0x7f93b015e160 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93a7fff700 1 -- 172.21.15.89:0/151624821 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac011990 con 0x7f93b015e160 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.812+0000 7f93a7fff700 1 -- 172.21.15.89:0/151624821 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac011d20 con 0x7f93b015e160 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.813+0000 7f93b72d4700 1 -- 172.21.15.89:0/151624821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b015e160 msgr2=0x7f93b00b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.813+0000 7f93b72d4700 1 --2- 172.21.15.89:0/151624821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b015e160 0x7f93b00b5da0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009840 tx=0x7f93ac009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.813+0000 7f93b72d4700 1 -- 172.21.15.89:0/151624821 shutdown_connections 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.813+0000 7f93b72d4700 1 --2- 172.21.15.89:0/151624821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b015e160 0x7f93b00b5da0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.813+0000 7f93b72d4700 1 --2- 172.21.15.89:0/151624821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b00afc90 0x7f93b0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.813+0000 7f93b72d4700 1 -- 172.21.15.89:0/151624821 >> 172.21.15.89:0/151624821 conn(0x7f93b00d0bf0 msgr2=0x7f93b00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:45.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.813+0000 7f93b72d4700 1 -- 172.21.15.89:0/151624821 shutdown_connections 2023-11-24T06:19:45.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.813+0000 7f93b72d4700 1 -- 172.21.15.89:0/151624821 wait complete. 2023-11-24T06:19:45.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.814+0000 7f93b72d4700 1 Processor -- start 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.814+0000 7f93b72d4700 1 -- start start 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b72d4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b00744d0 0x7f93b00c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b72d4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b00afc90 0x7f93b014b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b72d4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b015e160 0x7f93b0160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b72d4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b00c76b0 con 0x7f93b00afc90 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b72d4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b00c6aa0 con 0x7f93b00744d0 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b72d4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b00c6d80 con 0x7f93b015e160 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b486f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b00afc90 0x7f93b014b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b486f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b00afc90 0x7f93b014b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36886/0 (socket says 172.21.15.89:36886) 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b486f700 1 -- 172.21.15.89:0/3888408362 learned_addr learned my addr 172.21.15.89:0/3888408362 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b486f700 1 -- 172.21.15.89:0/3888408362 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b015e160 msgr2=0x7f93b0160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b486f700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b015e160 0x7f93b0160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b486f700 1 -- 172.21.15.89:0/3888408362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b00744d0 msgr2=0x7f93b00c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b486f700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b00744d0 0x7f93b00c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.815+0000 7f93b486f700 1 -- 172.21.15.89:0/3888408362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac0094a0 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.816+0000 7f93b486f700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b00afc90 0x7f93b014b580 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f93a000b530 tx=0x7f93a000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.816+0000 7f93a5ffb700 1 -- 172.21.15.89:0/3888408362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0011830 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.816+0000 7f93a5ffb700 1 -- 172.21.15.89:0/3888408362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a00119b0 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.816+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b00c7060 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.816+0000 7f93a5ffb700 1 -- 172.21.15.89:0/3888408362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0011d60 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.816+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b014bac0 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.817+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b0072bd0 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.817+0000 7f93a5ffb700 1 -- 172.21.15.89:0/3888408362 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f93a001a400 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.818+0000 7f93a5ffb700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f939406aed0 0x7f939406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.818+0000 7f93a5ffb700 1 -- 172.21.15.89:0/3888408362 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93a00896d0 con 0x7f93b00afc90 2023-11-24T06:19:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.818+0000 7f93b5070700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f939406aed0 0x7f939406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:45.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.818+0000 7f93b5070700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f939406aed0 0x7f939406d390 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f93ac000c00 tx=0x7f93ac010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:45.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:45.822+0000 7f93a5ffb700 1 -- 172.21.15.89:0/3888408362 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93a0054770 con 0x7f93b00afc90 2023-11-24T06:19:46.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.057+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f93b0074e60 con 0x7f93b00afc90 2023-11-24T06:19:46.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.988+0000 7f93a5ffb700 1 -- 172.21.15.89:0/3888408362 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f93a0057d90 con 0x7f93b00afc90 2023-11-24T06:19:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f939406aed0 msgr2=0x7f939406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f939406aed0 0x7f939406d390 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f93ac000c00 tx=0x7f93ac010000 comp rx=0 tx=0).stop 2023-11-24T06:19:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b00afc90 msgr2=0x7f93b014b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b00afc90 0x7f93b014b580 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f93a000b530 tx=0x7f93a000b8f0 comp rx=0 tx=0).stop 2023-11-24T06:19:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 shutdown_connections 2023-11-24T06:19:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b015e160 0x7f93b0160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f939406aed0 0x7f939406d390 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b00744d0 0x7f93b00c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 --2- 172.21.15.89:0/3888408362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b00afc90 0x7f93b014b580 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 >> 172.21.15.89:0/3888408362 conn(0x7f93b00d0bf0 msgr2=0x7f93b00b18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 shutdown_connections 2023-11-24T06:19:46.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:46.991+0000 7f93b72d4700 1 -- 172.21.15.89:0/3888408362 wait complete. 2023-11-24T06:19:47.009 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T06:19:47.009 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T06:19:47.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.134+0000 7f1fbf627700 1 Processor -- start 2023-11-24T06:19:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbf627700 1 -- start start 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbf627700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1fb815b370 0x7f1fb815b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbf627700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1fb80b3c60 0x7f1fb80b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbf627700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb80ae5e0 con 0x7f1fb81587f0 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbf627700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb80b4630 con 0x7f1fb815b370 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbf627700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb815be30 con 0x7f1fb80b3c60 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbcbc2700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1fb815b370 0x7f1fb815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbd3c3700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1fb81587f0 0x7f1fb806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40396/0 (socket says 172.21.15.89:40396) 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.135+0000 7f1fbd3c3700 1 -- 172.21.15.89:0/109642516 learned_addr learned my addr 172.21.15.89:0/109642516 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1fbdbc4700 1 --2- 172.21.15.89:0/109642516 >> v2:172.21.15.123:3300/0 conn(0x7f1fb80b3c60 0x7f1fb80b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1fbcbc2700 1 -- 172.21.15.89:0/109642516 >> v2:172.21.15.123:3300/0 conn(0x7f1fb80b3c60 msgr2=0x7f1fb80b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1fbcbc2700 1 --2- 172.21.15.89:0/109642516 >> v2:172.21.15.123:3300/0 conn(0x7f1fb80b3c60 0x7f1fb80b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1fbcbc2700 1 -- 172.21.15.89:0/109642516 >> v1:172.21.15.89:6789/0 conn(0x7f1fb81587f0 legacy=0x7f1fb806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1fbcbc2700 1 -- 172.21.15.89:0/109642516 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb815c110 con 0x7f1fb815b370 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1faffff700 1 -- 172.21.15.89:0/109642516 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659764100 0 0) 0x7f1fb80ae5e0 con 0x7f1fb81587f0 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1fbcbc2700 1 --2- 172.21.15.89:0/109642516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb815b800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009840 tx=0x7f1fb4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3281a5e2715d9214 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1faffff700 1 -- 172.21.15.89:0/109642516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4011830 con 0x7f1fb815b370 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1faffff700 1 -- 172.21.15.89:0/109642516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4011990 con 0x7f1fb815b370 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.136+0000 7f1faffff700 1 -- 172.21.15.89:0/109642516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4011d20 con 0x7f1fb815b370 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.137+0000 7f1fbf627700 1 -- 172.21.15.89:0/109642516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 msgr2=0x7f1fb815b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.137+0000 7f1fbf627700 1 --2- 172.21.15.89:0/109642516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb815b800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009840 tx=0x7f1fb4009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.137+0000 7f1fbf627700 1 -- 172.21.15.89:0/109642516 shutdown_connections 2023-11-24T06:19:47.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.137+0000 7f1fbf627700 1 --2- 172.21.15.89:0/109642516 >> v2:172.21.15.123:3300/0 conn(0x7f1fb80b3c60 0x7f1fb80b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.137+0000 7f1fbf627700 1 --2- 172.21.15.89:0/109642516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb815b800 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.137+0000 7f1fbf627700 1 -- 172.21.15.89:0/109642516 >> 172.21.15.89:0/109642516 conn(0x7f1fb80d0bf0 msgr2=0x7f1fb80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:47.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.137+0000 7f1fbf627700 1 -- 172.21.15.89:0/109642516 shutdown_connections 2023-11-24T06:19:47.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.137+0000 7f1fbf627700 1 -- 172.21.15.89:0/109642516 wait complete. 2023-11-24T06:19:47.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.138+0000 7f1fbf627700 1 Processor -- start 2023-11-24T06:19:47.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.138+0000 7f1fbf627700 1 -- start start 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbf627700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb80b3c60 0x7f1fb809c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbf627700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1fb81587f0 0x7f1fb809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbf627700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb814f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbf627700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb815e360 con 0x7f1fb815b370 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbf627700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb809bdf0 con 0x7f1fb80b3c60 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbf627700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb809af40 con 0x7f1fb81587f0 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb814f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb814f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36902/0 (socket says 172.21.15.89:36902) 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 -- 172.21.15.89:0/155479686 learned_addr learned my addr 172.21.15.89:0/155479686 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 -- 172.21.15.89:0/155479686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1fb81587f0 msgr2=0x7f1fb809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1fb81587f0 0x7f1fb809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbd3c3700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb80b3c60 0x7f1fb809c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 -- 172.21.15.89:0/155479686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb80b3c60 msgr2=0x7f1fb809c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb80b3c60 0x7f1fb809c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 -- 172.21.15.89:0/155479686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb40094a0 con 0x7f1fb815b370 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbdbc4700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb814f7f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0007e40 tx=0x7f1fb000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fadffb700 1 -- 172.21.15.89:0/155479686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb000eea0 con 0x7f1fb815b370 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fadffb700 1 -- 172.21.15.89:0/155479686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb0014420 con 0x7f1fb815b370 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb809b1e0 con 0x7f1fb815b370 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.139+0000 7f1fadffb700 1 -- 172.21.15.89:0/155479686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb00147b0 con 0x7f1fb815b370 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.140+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb809b7a0 con 0x7f1fb815b370 2023-11-24T06:19:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.141+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb8068fb0 con 0x7f1fb815b370 2023-11-24T06:19:47.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.141+0000 7f1fadffb700 1 -- 172.21.15.89:0/155479686 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1fb0014910 con 0x7f1fb815b370 2023-11-24T06:19:47.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.141+0000 7f1fadffb700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f9c06aed0 0x7f1f9c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:47.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.141+0000 7f1fadffb700 1 -- 172.21.15.89:0/155479686 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fb0089e20 con 0x7f1fb815b370 2023-11-24T06:19:47.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.141+0000 7f1fbd3c3700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f9c06aed0 0x7f1f9c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:47.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.142+0000 7f1fbd3c3700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f9c06aed0 0x7f1f9c06d390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8000f60 tx=0x7f1fa8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:47.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.146+0000 7f1fadffb700 1 -- 172.21.15.89:0/155479686 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fb0054ed0 con 0x7f1fb815b370 2023-11-24T06:19:47.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.378+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1fb815e4a0 con 0x7f1fb815b370 2023-11-24T06:19:47.925 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:19:47.928 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:19:47.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.991+0000 7f1fadffb700 1 -- 172.21.15.89:0/155479686 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1fb00584f0 con 0x7f1fb815b370 2023-11-24T06:19:47.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f9c06aed0 msgr2=0x7f1f9c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:47.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f9c06aed0 0x7f1f9c06d390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8000f60 tx=0x7f1fa8009040 comp rx=0 tx=0).stop 2023-11-24T06:19:47.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 msgr2=0x7f1fb814f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:47.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb814f7f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0007e40 tx=0x7f1fb000e3c0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 shutdown_connections 2023-11-24T06:19:47.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fb815b370 0x7f1fb814f7f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1f9c06aed0 0x7f1f9c06d390 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb80b3c60 0x7f1fb809c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 --2- 172.21.15.89:0/155479686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1fb81587f0 0x7f1fb809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:47.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 >> 172.21.15.89:0/155479686 conn(0x7f1fb80d0bf0 msgr2=0x7f1fb80aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:47.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 shutdown_connections 2023-11-24T06:19:47.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:47.994+0000 7f1fbf627700 1 -- 172.21.15.89:0/155479686 wait complete. 2023-11-24T06:19:48.009 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T06:19:48.009 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:48.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.167+0000 7f923bbf9700 1 Processor -- start 2023-11-24T06:19:48.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.167+0000 7f923bbf9700 1 -- start start 2023-11-24T06:19:48.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f923bbf9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f923415bdd0 0x7f92341604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f923bbf9700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92340b3370 0x7f92340b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f923bbf9700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92340b2030 con 0x7f92340d1c70 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f923bbf9700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340b2f50 con 0x7f92340b3370 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f923bbf9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340b22d0 con 0x7f923415bdd0 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f9239194700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92340b3370 0x7f92340b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f923a196700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f92340d1c70 0x7f92340ce6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40412/0 (socket says 172.21.15.89:40412) 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f923a196700 1 -- 172.21.15.89:0/2105467342 learned_addr learned my addr 172.21.15.89:0/2105467342 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f9239995700 1 --2- 172.21.15.89:0/2105467342 >> v2:172.21.15.123:3300/0 conn(0x7f923415bdd0 0x7f92341604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f9239194700 1 -- 172.21.15.89:0/2105467342 >> v2:172.21.15.123:3300/0 conn(0x7f923415bdd0 msgr2=0x7f92341604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f9239194700 1 --2- 172.21.15.89:0/2105467342 >> v2:172.21.15.123:3300/0 conn(0x7f923415bdd0 0x7f92341604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f9239194700 1 -- 172.21.15.89:0/2105467342 >> v1:172.21.15.89:6789/0 conn(0x7f92340d1c70 legacy=0x7f92340ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f9239194700 1 -- 172.21.15.89:0/2105467342 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92340b25b0 con 0x7f92340b3370 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f9238993700 1 -- 172.21.15.89:0/2105467342 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993554708 0 0) 0x7f92340b2030 con 0x7f92340d1c70 2023-11-24T06:19:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.168+0000 7f9239995700 1 --2- 172.21.15.89:0/2105467342 >> v2:172.21.15.123:3300/0 conn(0x7f923415bdd0 0x7f92341604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f9239194700 1 --2- 172.21.15.89:0/2105467342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340b3370 0x7f92340b1af0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f923406ec10 tx=0x7f9230009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c66531d1254239 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f9238993700 1 -- 172.21.15.89:0/2105467342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230011830 con 0x7f92340b3370 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f9238993700 1 -- 172.21.15.89:0/2105467342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9230011990 con 0x7f92340b3370 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f9238993700 1 -- 172.21.15.89:0/2105467342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230011d20 con 0x7f92340b3370 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f923bbf9700 1 -- 172.21.15.89:0/2105467342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340b3370 msgr2=0x7f92340b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f923bbf9700 1 --2- 172.21.15.89:0/2105467342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340b3370 0x7f92340b1af0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f923406ec10 tx=0x7f9230009b20 comp rx=0 tx=0).stop 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f923bbf9700 1 -- 172.21.15.89:0/2105467342 shutdown_connections 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f923bbf9700 1 --2- 172.21.15.89:0/2105467342 >> v2:172.21.15.123:3300/0 conn(0x7f923415bdd0 0x7f92341604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f923bbf9700 1 --2- 172.21.15.89:0/2105467342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340b3370 0x7f92340b1af0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f923bbf9700 1 -- 172.21.15.89:0/2105467342 >> 172.21.15.89:0/2105467342 conn(0x7f92340731a0 msgr2=0x7f923409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f923bbf9700 1 -- 172.21.15.89:0/2105467342 shutdown_connections 2023-11-24T06:19:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.169+0000 7f923bbf9700 1 -- 172.21.15.89:0/2105467342 wait complete. 2023-11-24T06:19:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.170+0000 7f923bbf9700 1 Processor -- start 2023-11-24T06:19:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.170+0000 7f923bbf9700 1 -- start start 2023-11-24T06:19:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f923bbf9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92340b3370 0x7f923414d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f923bbf9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340d1c70 0x7f92340c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f923bbf9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f923415bdd0 0x7f92340ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f923bbf9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92340caa10 con 0x7f92340d1c70 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f923bbf9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92340c9ce0 con 0x7f923415bdd0 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f923bbf9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92340c9440 con 0x7f92340b3370 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f9239194700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340d1c70 0x7f92340c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f9239194700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340d1c70 0x7f92340c8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36912/0 (socket says 172.21.15.89:36912) 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f9239194700 1 -- 172.21.15.89:0/1695219915 learned_addr learned my addr 172.21.15.89:0/1695219915 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f9239194700 1 -- 172.21.15.89:0/1695219915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92340b3370 msgr2=0x7f923414d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.171+0000 7f9239194700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92340b3370 0x7f923414d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f9239194700 1 -- 172.21.15.89:0/1695219915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f923415bdd0 msgr2=0x7f92340ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f9239194700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f923415bdd0 0x7f92340ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f9239194700 1 -- 172.21.15.89:0/1695219915 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92300094a0 con 0x7f92340d1c70 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f9239194700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340d1c70 0x7f92340c8db0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9230000c00 tx=0x7f9230004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f922a7fc700 1 -- 172.21.15.89:0/1695219915 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92300038a0 con 0x7f92340d1c70 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f922a7fc700 1 -- 172.21.15.89:0/1695219915 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923001f070 con 0x7f92340d1c70 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92340c9720 con 0x7f92340d1c70 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f922a7fc700 1 -- 172.21.15.89:0/1695219915 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230003c60 con 0x7f92340d1c70 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.172+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92340b4940 con 0x7f92340d1c70 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.173+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92340b4ad0 con 0x7f92340d1c70 2023-11-24T06:19:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.173+0000 7f922a7fc700 1 -- 172.21.15.89:0/1695219915 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f923001a430 con 0x7f92340d1c70 2023-11-24T06:19:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.174+0000 7f922a7fc700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f921806aed0 0x7f921806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.174+0000 7f922a7fc700 1 -- 172.21.15.89:0/1695219915 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9230021070 con 0x7f92340d1c70 2023-11-24T06:19:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.174+0000 7f9239995700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f921806aed0 0x7f921806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.174+0000 7f9239995700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f921806aed0 0x7f921806d390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f922400a8b0 tx=0x7f9224008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.178+0000 7f922a7fc700 1 -- 172.21.15.89:0/1695219915 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9230056150 con 0x7f92340d1c70 2023-11-24T06:19:48.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.408+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9234067f60 con 0x7f92340d1c70 2023-11-24T06:19:48.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.409+0000 7f922a7fc700 1 -- 172.21.15.89:0/1695219915 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f9230015090 con 0x7f92340d1c70 2023-11-24T06:19:48.412 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:48.412 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/503786989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":503786989},{"type":"v1","addr":"172.21.15.123:6834","nonce":503786989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3511736853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3511736853},{"type":"v1","addr":"172.21.15.89:6836","nonce":3511736853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5091,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053513777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2053513777},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053513777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:19:43.989850+0000","modified":"2023-11-24T06:19:47.380732+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6835/1601724762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1601724762},{"type":"v1","addr":"172.21.15.123:6835","nonce":1601724762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:19:48.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f921806aed0 msgr2=0x7f921806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f921806aed0 0x7f921806d390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f922400a8b0 tx=0x7f9224008040 comp rx=0 tx=0).stop 2023-11-24T06:19:48.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340d1c70 msgr2=0x7f92340c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340d1c70 0x7f92340c8db0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9230000c00 tx=0x7f9230004750 comp rx=0 tx=0).stop 2023-11-24T06:19:48.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 shutdown_connections 2023-11-24T06:19:48.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f921806aed0 0x7f921806d390 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f923415bdd0 0x7f92340ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92340b3370 0x7f923414d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 --2- 172.21.15.89:0/1695219915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92340d1c70 0x7f92340c8db0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 >> 172.21.15.89:0/1695219915 conn(0x7f92340731a0 msgr2=0x7f923409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:48.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 shutdown_connections 2023-11-24T06:19:48.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.411+0000 7f923bbf9700 1 -- 172.21.15.89:0/1695219915 wait complete. 2023-11-24T06:19:48.415 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2023-11-24T06:19:48.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:19:48.475 INFO:tasks.ceph.mds.b.smithi123.stderr:2023-11-24T06:19:48.472+0000 7ff35c4d8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:19:48.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.543+0000 7f949c7f6700 1 Processor -- start 2023-11-24T06:19:48.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.543+0000 7f949c7f6700 1 -- start start 2023-11-24T06:19:48.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.544+0000 7f949c7f6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 0x7f949415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.544+0000 7f949c7f6700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f949406dc20 0x7f9494073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.544+0000 7f949c7f6700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949415eb00 con 0x7f9494147780 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.544+0000 7f949c7f6700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f949415ede0 con 0x7f949406dc20 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.544+0000 7f949c7f6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949415f0c0 con 0x7f94941473a0 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.544+0000 7f9499d91700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9494147780 0x7f949406c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40424/0 (socket says 172.21.15.89:40424) 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.544+0000 7f9499d91700 1 -- 172.21.15.89:0/223577289 learned_addr learned my addr 172.21.15.89:0/223577289 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f949ad93700 1 --2- 172.21.15.89:0/223577289 >> v2:172.21.15.123:3300/0 conn(0x7f949406dc20 0x7f9494073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421505396 0 0) 0x7f949415eb00 con 0x7f9494147780 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f949a592700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 0x7f949415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94800035b0 con 0x7f9494147780 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665709737 0 0) 0x7f94800035b0 con 0x7f9494147780 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 msgr2=0x7f949415e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 0x7f949415e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 >> v2:172.21.15.123:3300/0 conn(0x7f949406dc20 msgr2=0x7f9494073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 --2- 172.21.15.89:0/223577289 >> v2:172.21.15.123:3300/0 conn(0x7f949406dc20 0x7f9494073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949415f3a0 con 0x7f9494147780 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f9490001670 con 0x7f9494147780 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f949ad93700 1 --2- 172.21.15.89:0/223577289 >> v2:172.21.15.123:3300/0 conn(0x7f949406dc20 0x7f9494073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f949a592700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 0x7f949415e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 >> v1:172.21.15.89:6789/0 conn(0x7f9494147780 legacy=0x7f949406c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480003a00 0x7f9480003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9480004850 0x7f9480004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 0x7f9480005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949415ede0 con 0x7f9480004850 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949415f0c0 con 0x7f9480003a00 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948000ba90 con 0x7f94941473a0 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499d91700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9480004850 0x7f9480004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f949ad93700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 0x7f9480005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f949a592700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480003a00 0x7f9480003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499d91700 1 -- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 msgr2=0x7f9480005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499d91700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 0x7f9480005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499d91700 1 -- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480003a00 msgr2=0x7f9480003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499d91700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480003a00 0x7f9480003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499d91700 1 -- 172.21.15.89:0/223577289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94800035b0 con 0x7f9480004850 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f949ad93700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94941473a0 0x7f9480005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.545+0000 7f9499d91700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9480004850 0x7f9480004cd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f949000c2f0 tx=0x7f9480005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5fe4f79be48c6d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949000ce30 con 0x7f9480004850 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9490005960 con 0x7f9480004850 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f9499590700 1 -- 172.21.15.89:0/223577289 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490005d10 con 0x7f9480004850 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f949c7f6700 1 -- 172.21.15.89:0/223577289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9480004850 msgr2=0x7f9480004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f949c7f6700 1 --2- 172.21.15.89:0/223577289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9480004850 0x7f9480004cd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f949000c2f0 tx=0x7f9480005880 comp rx=0 tx=0).stop 2023-11-24T06:19:48.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f949ad93700 1 -- 172.21.15.89:0/223577289 reap_dead start 2023-11-24T06:19:48.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f949c7f6700 1 -- 172.21.15.89:0/223577289 shutdown_connections 2023-11-24T06:19:48.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f949c7f6700 1 -- 172.21.15.89:0/223577289 >> 172.21.15.89:0/223577289 conn(0x7f94940b3130 msgr2=0x7f94940d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:48.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f949c7f6700 1 -- 172.21.15.89:0/223577289 shutdown_connections 2023-11-24T06:19:48.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.546+0000 7f949c7f6700 1 -- 172.21.15.89:0/223577289 wait complete. 2023-11-24T06:19:48.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.547+0000 7f949c7f6700 1 Processor -- start 2023-11-24T06:19:48.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.547+0000 7f949c7f6700 1 -- start start 2023-11-24T06:19:48.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949c7f6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480004850 0x7f949415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949c7f6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9494073f90 0x7f9494074410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949c7f6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94940ad9b0 0x7f94940ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949c7f6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94940af310 con 0x7f94940ad9b0 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949c7f6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9494074950 con 0x7f9480004850 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949c7f6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94940ae690 con 0x7f9494073f90 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949ad93700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94940ad9b0 0x7f94940ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949ad93700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94940ad9b0 0x7f94940ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36934/0 (socket says 172.21.15.89:36934) 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949ad93700 1 -- 172.21.15.89:0/3677158687 learned_addr learned my addr 172.21.15.89:0/3677158687 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f949a592700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480004850 0x7f949415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.548+0000 7f9499d91700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9494073f90 0x7f9494074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f949ad93700 1 -- 172.21.15.89:0/3677158687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9494073f90 msgr2=0x7f9494074410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f949ad93700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9494073f90 0x7f9494074410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f949ad93700 1 -- 172.21.15.89:0/3677158687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480004850 msgr2=0x7f949415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f949ad93700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480004850 0x7f949415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f949ad93700 1 -- 172.21.15.89:0/3677158687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940ae970 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f949ad93700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94940ad9b0 0x7f94940ade30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f94940b3830 tx=0x7f948c0048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f948affd700 1 -- 172.21.15.89:0/3677158687 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c004930 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f948affd700 1 -- 172.21.15.89:0/3677158687 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948c00f050 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f948affd700 1 -- 172.21.15.89:0/3677158687 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c007dc0 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94940aec50 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.549+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94940ae370 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.550+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9494067f60 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.550+0000 7f948affd700 1 -- 172.21.15.89:0/3677158687 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f948c01e460 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.551+0000 7f948affd700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f947806aed0 0x7f947806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.551+0000 7f948affd700 1 -- 172.21.15.89:0/3677158687 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f948c089860 con 0x7f94940ad9b0 2023-11-24T06:19:48.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.551+0000 7f949a592700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f947806aed0 0x7f947806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.551+0000 7f949a592700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f947806aed0 0x7f947806d390 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f94840044c0 tx=0x7f9484008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.555+0000 7f948affd700 1 -- 172.21.15.89:0/3677158687 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f948c054910 con 0x7f94940ad9b0 2023-11-24T06:19:48.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.746+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f949406c240 con 0x7f94940ad9b0 2023-11-24T06:19:48.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.747+0000 7f948affd700 1 -- 172.21.15.89:0/3677158687 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f948c057f30 con 0x7f94940ad9b0 2023-11-24T06:19:48.751 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:48.751 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":34,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:19:45.595335+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f947806aed0 msgr2=0x7f947806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f947806aed0 0x7f947806d390 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f94840044c0 tx=0x7f9484008040 comp rx=0 tx=0).stop 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94940ad9b0 msgr2=0x7f94940ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94940ad9b0 0x7f94940ade30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f94940b3830 tx=0x7f948c0048c0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 shutdown_connections 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94940ad9b0 0x7f94940ade30 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f947806aed0 0x7f947806d390 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9480004850 0x7f949415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 --2- 172.21.15.89:0/3677158687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9494073f90 0x7f9494074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.749+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 >> 172.21.15.89:0/3677158687 conn(0x7f94940b3130 msgr2=0x7f9494145530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:48.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.750+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 shutdown_connections 2023-11-24T06:19:48.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.750+0000 7f949c7f6700 1 -- 172.21.15.89:0/3677158687 wait complete. 2023-11-24T06:19:48.765 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T06:19:48.765 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:19:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.881+0000 7f3b6db62700 1 Processor -- start 2023-11-24T06:19:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.881+0000 7f3b6db62700 1 -- start start 2023-11-24T06:19:48.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b6db62700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b6806c710 0x7f3b680b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b6db62700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3b680af4d0 0x7f3b680afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b6db62700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b680ae430 con 0x7f3b6815e080 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b6db62700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b680af0b0 con 0x7f3b6806c710 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b6db62700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b680b05a0 con 0x7f3b680af4d0 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b66ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b6806c710 0x7f3b680b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b66ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b6806c710 0x7f3b680b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36940/0 (socket says 172.21.15.89:36940) 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b66ffd700 1 -- 172.21.15.89:0/167940243 learned_addr learned my addr 172.21.15.89:0/167940243 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b67fff700 1 --2- 172.21.15.89:0/167940243 >> v2:172.21.15.123:3300/0 conn(0x7f3b680af4d0 0x7f3b680afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b66ffd700 1 -- 172.21.15.89:0/167940243 >> v2:172.21.15.123:3300/0 conn(0x7f3b680af4d0 msgr2=0x7f3b680afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.882+0000 7f3b66ffd700 1 --2- 172.21.15.89:0/167940243 >> v2:172.21.15.123:3300/0 conn(0x7f3b680af4d0 0x7f3b680afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b66ffd700 1 -- 172.21.15.89:0/167940243 >> v1:172.21.15.123:6789/0 conn(0x7f3b6815e080 legacy=0x7f3b6806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b66ffd700 1 -- 172.21.15.89:0/167940243 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b680ae710 con 0x7f3b6806c710 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b66ffd700 1 --2- 172.21.15.89:0/167940243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b680b5050 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3b6809eea0 tx=0x7f3b58009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f760b9c48109102f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b667fc700 1 -- 172.21.15.89:0/167940243 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b58004230 con 0x7f3b6806c710 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b667fc700 1 -- 172.21.15.89:0/167940243 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b58004390 con 0x7f3b6806c710 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b667fc700 1 -- 172.21.15.89:0/167940243 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580036a0 con 0x7f3b6806c710 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b6db62700 1 -- 172.21.15.89:0/167940243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 msgr2=0x7f3b680b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b6db62700 1 --2- 172.21.15.89:0/167940243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b680b5050 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3b6809eea0 tx=0x7f3b58009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b6db62700 1 -- 172.21.15.89:0/167940243 shutdown_connections 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b6db62700 1 --2- 172.21.15.89:0/167940243 >> v2:172.21.15.123:3300/0 conn(0x7f3b680af4d0 0x7f3b680afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b6db62700 1 --2- 172.21.15.89:0/167940243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b680b5050 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b6db62700 1 -- 172.21.15.89:0/167940243 >> 172.21.15.89:0/167940243 conn(0x7f3b68071b90 msgr2=0x7f3b68071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b6db62700 1 -- 172.21.15.89:0/167940243 shutdown_connections 2023-11-24T06:19:48.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.883+0000 7f3b6db62700 1 -- 172.21.15.89:0/167940243 wait complete. 2023-11-24T06:19:48.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.885+0000 7f3b6db62700 1 Processor -- start 2023-11-24T06:19:48.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.885+0000 7f3b6db62700 1 -- start start 2023-11-24T06:19:48.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.885+0000 7f3b6db62700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b68141bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.885+0000 7f3b6db62700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b680af4d0 0x7f3b681607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.885+0000 7f3b6db62700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b6815e080 0x7f3b68153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.885+0000 7f3b6db62700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b681420f0 con 0x7f3b6806c710 2023-11-24T06:19:48.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.885+0000 7f3b6db62700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b681570c0 con 0x7f3b6815e080 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.885+0000 7f3b6db62700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68142e60 con 0x7f3b680af4d0 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b68141bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b68141bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36944/0 (socket says 172.21.15.89:36944) 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 -- 172.21.15.89:0/1858591160 learned_addr learned my addr 172.21.15.89:0/1858591160 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b67fff700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b6815e080 0x7f3b68153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b66ffd700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b680af4d0 0x7f3b681607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 -- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b680af4d0 msgr2=0x7f3b681607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b680af4d0 0x7f3b681607f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 -- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b6815e080 msgr2=0x7f3b68153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b6815e080 0x7f3b68153d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 -- 172.21.15.89:0/1858591160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b58009550 con 0x7f3b6806c710 2023-11-24T06:19:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b66ffd700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b680af4d0 0x7f3b681607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b677fe700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b68141bb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000b4a0 tx=0x7f3b5000b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b5ffff700 1 -- 172.21.15.89:0/1858591160 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50011830 con 0x7f3b6806c710 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b5ffff700 1 -- 172.21.15.89:0/1858591160 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b50011990 con 0x7f3b6806c710 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b5ffff700 1 -- 172.21.15.89:0/1858591160 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50011d40 con 0x7f3b6806c710 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b68156b00 con 0x7f3b6806c710 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.886+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6815fff0 con 0x7f3b6806c710 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.887+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b680753f0 con 0x7f3b6806c710 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.888+0000 7f3b5ffff700 1 -- 172.21.15.89:0/1858591160 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3b5001a400 con 0x7f3b6806c710 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.888+0000 7f3b5ffff700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b4806aed0 0x7f3b4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.888+0000 7f3b5ffff700 1 -- 172.21.15.89:0/1858591160 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b50089270 con 0x7f3b6806c710 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.888+0000 7f3b66ffd700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b4806aed0 0x7f3b4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:48.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.888+0000 7f3b66ffd700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b4806aed0 0x7f3b4806d390 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b58008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:48.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:48.892+0000 7f3b5ffff700 1 -- 172.21.15.89:0/1858591160 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b50054320 con 0x7f3b6806c710 2023-11-24T06:19:49.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.078+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b680aecd0 con 0x7f3b4806aed0 2023-11-24T06:19:49.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.086+0000 7f3b5ffff700 1 -- 172.21.15.89:0/1858591160 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289469 (secure 0 0 0) 0x7f3b58003850 con 0x7f3b4806aed0 2023-11-24T06:19:49.094 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:49.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b4806aed0 msgr2=0x7f3b4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b4806aed0 0x7f3b4806d390 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b58008030 comp rx=0 tx=0).stop 2023-11-24T06:19:49.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 msgr2=0x7f3b68141bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b68141bb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000b4a0 tx=0x7f3b5000b7b0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 shutdown_connections 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b4806aed0 0x7f3b4806d390 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b6815e080 0x7f3b68153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b6806c710 0x7f3b68141bb0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 --2- 172.21.15.89:0/1858591160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b680af4d0 0x7f3b681607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 >> 172.21.15.89:0/1858591160 conn(0x7f3b68071b90 msgr2=0x7f3b6806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 shutdown_connections 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.089+0000 7f3b6db62700 1 -- 172.21.15.89:0/1858591160 wait complete. 2023-11-24T06:19:49.097 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:19:49.104 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2023-11-24T06:19:47.731509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220384,"kb_used_data":4196,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754336,"statfs":{"total":773094113280,"available":772868440064,"internally_reserved":0,"allocated":4296704,"data_stored":2375656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.146883"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624830+0000","last_change":"2023-11-24T06:19:44.620576+0000","last_active":"2023-11-24T06:19:44.624830+0000","last_peered":"2023-11-24T06:19:44.624830+0000","last_clean":"2023-11-24T06:19:44.624830+0000","last_became_active":"2023-11-24T06:19:44.620261+0000","last_became_peered":"2023-11-24T06:19:44.620261+0000","last_unstale":"2023-11-24T06:19:44.624830+0000","last_undegraded":"2023-11-24T06:19:44.624830+0000","last_fullsized":"2023-11-24T06:19:44.624830+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:07.005490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604933+0000","last_change":"2023-11-24T06:19:44.617073+0000","last_active":"2023-11-24T06:19:45.604933+0000","last_peered":"2023-11-24T06:19:45.604933+0000","last_clean":"2023-11-24T06:19:45.604933+0000","last_became_active":"2023-11-24T06:19:44.616814+0000","last_became_peered":"2023-11-24T06:19:44.616814+0000","last_unstale":"2023-11-24T06:19:45.604933+0000","last_undegraded":"2023-11-24T06:19:45.604933+0000","last_fullsized":"2023-11-24T06:19:45.604933+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:22:35.277438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603717+0000","last_change":"2023-11-24T06:19:44.620975+0000","last_active":"2023-11-24T06:19:45.603717+0000","last_peered":"2023-11-24T06:19:45.603717+0000","last_clean":"2023-11-24T06:19:45.603717+0000","last_became_active":"2023-11-24T06:19:44.620500+0000","last_became_peered":"2023-11-24T06:19:44.620500+0000","last_unstale":"2023-11-24T06:19:45.603717+0000","last_undegraded":"2023-11-24T06:19:45.603717+0000","last_fullsized":"2023-11-24T06:19:45.603717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:22.578783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106954+0000","last_change":"2023-11-24T06:19:44.620120+0000","last_active":"2023-11-24T06:19:46.106954+0000","last_peered":"2023-11-24T06:19:46.106954+0000","last_clean":"2023-11-24T06:19:46.106954+0000","last_became_active":"2023-11-24T06:19:44.619797+0000","last_became_peered":"2023-11-24T06:19:44.619797+0000","last_unstale":"2023-11-24T06:19:46.106954+0000","last_undegraded":"2023-11-24T06:19:46.106954+0000","last_fullsized":"2023-11-24T06:19:46.106954+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:09:27.389299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144468+0000","last_change":"2023-11-24T06:19:44.607813+0000","last_active":"2023-11-24T06:19:46.144468+0000","last_peered":"2023-11-24T06:19:46.144468+0000","last_clean":"2023-11-24T06:19:46.144468+0000","last_became_active":"2023-11-24T06:19:44.607521+0000","last_became_peered":"2023-11-24T06:19:44.607521+0000","last_unstale":"2023-11-24T06:19:46.144468+0000","last_undegraded":"2023-11-24T06:19:46.144468+0000","last_fullsized":"2023-11-24T06:19:46.144468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:03.428940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604755+0000","last_change":"2023-11-24T06:19:44.617997+0000","last_active":"2023-11-24T06:19:45.604755+0000","last_peered":"2023-11-24T06:19:45.604755+0000","last_clean":"2023-11-24T06:19:45.604755+0000","last_became_active":"2023-11-24T06:19:44.617699+0000","last_became_peered":"2023-11-24T06:19:44.617699+0000","last_unstale":"2023-11-24T06:19:45.604755+0000","last_undegraded":"2023-11-24T06:19:45.604755+0000","last_fullsized":"2023-11-24T06:19:45.604755+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:17.149210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604218+0000","last_change":"2023-11-24T06:19:44.615653+0000","last_active":"2023-11-24T06:19:45.604218+0000","last_peered":"2023-11-24T06:19:45.604218+0000","last_clean":"2023-11-24T06:19:45.604218+0000","last_became_active":"2023-11-24T06:19:44.615411+0000","last_became_peered":"2023-11-24T06:19:44.615411+0000","last_unstale":"2023-11-24T06:19:45.604218+0000","last_undegraded":"2023-11-24T06:19:45.604218+0000","last_fullsized":"2023-11-24T06:19:45.604218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:54.283680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603821+0000","last_change":"2023-11-24T06:19:44.605475+0000","last_active":"2023-11-24T06:19:45.603821+0000","last_peered":"2023-11-24T06:19:45.603821+0000","last_clean":"2023-11-24T06:19:45.603821+0000","last_became_active":"2023-11-24T06:19:44.605100+0000","last_became_peered":"2023-11-24T06:19:44.605100+0000","last_unstale":"2023-11-24T06:19:45.603821+0000","last_undegraded":"2023-11-24T06:19:45.603821+0000","last_fullsized":"2023-11-24T06:19:45.603821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:22.578783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.621965+0000","last_change":"2023-11-24T06:19:44.615217+0000","last_active":"2023-11-24T06:19:44.621965+0000","last_peered":"2023-11-24T06:19:44.621965+0000","last_clean":"2023-11-24T06:19:44.621965+0000","last_became_active":"2023-11-24T06:19:44.613739+0000","last_became_peered":"2023-11-24T06:19:44.613739+0000","last_unstale":"2023-11-24T06:19:44.621965+0000","last_undegraded":"2023-11-24T06:19:44.621965+0000","last_fullsized":"2023-11-24T06:19:44.621965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:36.466491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107232+0000","last_change":"2023-11-24T06:19:44.614457+0000","last_active":"2023-11-24T06:19:46.107232+0000","last_peered":"2023-11-24T06:19:46.107232+0000","last_clean":"2023-11-24T06:19:46.107232+0000","last_became_active":"2023-11-24T06:19:44.614123+0000","last_became_peered":"2023-11-24T06:19:44.614123+0000","last_unstale":"2023-11-24T06:19:46.107232+0000","last_undegraded":"2023-11-24T06:19:46.107232+0000","last_fullsized":"2023-11-24T06:19:46.107232+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:11:48.592746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603724+0000","last_change":"2023-11-24T06:19:44.611487+0000","last_active":"2023-11-24T06:19:45.603724+0000","last_peered":"2023-11-24T06:19:45.603724+0000","last_clean":"2023-11-24T06:19:45.603724+0000","last_became_active":"2023-11-24T06:19:44.610852+0000","last_became_peered":"2023-11-24T06:19:44.610852+0000","last_unstale":"2023-11-24T06:19:45.603724+0000","last_undegraded":"2023-11-24T06:19:45.603724+0000","last_fullsized":"2023-11-24T06:19:45.603724+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:04:44.567943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603612+0000","last_change":"2023-11-24T06:19:44.621311+0000","last_active":"2023-11-24T06:19:45.603612+0000","last_peered":"2023-11-24T06:19:45.603612+0000","last_clean":"2023-11-24T06:19:45.603612+0000","last_became_active":"2023-11-24T06:19:44.619835+0000","last_became_peered":"2023-11-24T06:19:44.619835+0000","last_unstale":"2023-11-24T06:19:45.603612+0000","last_undegraded":"2023-11-24T06:19:45.603612+0000","last_fullsized":"2023-11-24T06:19:45.603612+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:04:44.567943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144597+0000","last_change":"2023-11-24T06:19:44.604646+0000","last_active":"2023-11-24T06:19:46.144597+0000","last_peered":"2023-11-24T06:19:46.144597+0000","last_clean":"2023-11-24T06:19:46.144597+0000","last_became_active":"2023-11-24T06:19:44.604275+0000","last_became_peered":"2023-11-24T06:19:44.604275+0000","last_unstale":"2023-11-24T06:19:46.144597+0000","last_undegraded":"2023-11-24T06:19:46.144597+0000","last_fullsized":"2023-11-24T06:19:46.144597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:57.374633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604704+0000","last_change":"2023-11-24T06:19:44.605526+0000","last_active":"2023-11-24T06:19:45.604704+0000","last_peered":"2023-11-24T06:19:45.604704+0000","last_clean":"2023-11-24T06:19:45.604704+0000","last_became_active":"2023-11-24T06:19:44.605255+0000","last_became_peered":"2023-11-24T06:19:44.605255+0000","last_unstale":"2023-11-24T06:19:45.604704+0000","last_undegraded":"2023-11-24T06:19:45.604704+0000","last_fullsized":"2023-11-24T06:19:45.604704+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:34.436870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603211+0000","last_change":"2023-11-24T06:19:44.621768+0000","last_active":"2023-11-24T06:19:45.603211+0000","last_peered":"2023-11-24T06:19:45.603211+0000","last_clean":"2023-11-24T06:19:45.603211+0000","last_became_active":"2023-11-24T06:19:44.621481+0000","last_became_peered":"2023-11-24T06:19:44.621481+0000","last_unstale":"2023-11-24T06:19:45.603211+0000","last_undegraded":"2023-11-24T06:19:45.603211+0000","last_fullsized":"2023-11-24T06:19:45.603211+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:17.543360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604002+0000","last_change":"2023-11-24T06:19:44.611740+0000","last_active":"2023-11-24T06:19:45.604002+0000","last_peered":"2023-11-24T06:19:45.604002+0000","last_clean":"2023-11-24T06:19:45.604002+0000","last_became_active":"2023-11-24T06:19:44.611350+0000","last_became_peered":"2023-11-24T06:19:44.611350+0000","last_unstale":"2023-11-24T06:19:45.604002+0000","last_undegraded":"2023-11-24T06:19:45.604002+0000","last_fullsized":"2023-11-24T06:19:45.604002+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:35:04.205238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603943+0000","last_change":"2023-11-24T06:19:44.619948+0000","last_active":"2023-11-24T06:19:45.603943+0000","last_peered":"2023-11-24T06:19:45.603943+0000","last_clean":"2023-11-24T06:19:45.603943+0000","last_became_active":"2023-11-24T06:19:44.619146+0000","last_became_peered":"2023-11-24T06:19:44.619146+0000","last_unstale":"2023-11-24T06:19:45.603943+0000","last_undegraded":"2023-11-24T06:19:45.603943+0000","last_fullsized":"2023-11-24T06:19:45.603943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:24.264848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605113+0000","last_change":"2023-11-24T06:19:44.610677+0000","last_active":"2023-11-24T06:19:45.605113+0000","last_peered":"2023-11-24T06:19:45.605113+0000","last_clean":"2023-11-24T06:19:45.605113+0000","last_became_active":"2023-11-24T06:19:44.609508+0000","last_became_peered":"2023-11-24T06:19:44.609508+0000","last_unstale":"2023-11-24T06:19:45.605113+0000","last_undegraded":"2023-11-24T06:19:45.605113+0000","last_fullsized":"2023-11-24T06:19:45.605113+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:26:06.485092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604804+0000","last_change":"2023-11-24T06:19:44.608764+0000","last_active":"2023-11-24T06:19:45.604804+0000","last_peered":"2023-11-24T06:19:45.604804+0000","last_clean":"2023-11-24T06:19:45.604804+0000","last_became_active":"2023-11-24T06:19:44.608472+0000","last_became_peered":"2023-11-24T06:19:44.608472+0000","last_unstale":"2023-11-24T06:19:45.604804+0000","last_undegraded":"2023-11-24T06:19:45.604804+0000","last_fullsized":"2023-11-24T06:19:45.604804+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:41:17.872131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144529+0000","last_change":"2023-11-24T06:19:44.617798+0000","last_active":"2023-11-24T06:19:46.144529+0000","last_peered":"2023-11-24T06:19:46.144529+0000","last_clean":"2023-11-24T06:19:46.144529+0000","last_became_active":"2023-11-24T06:19:44.617465+0000","last_became_peered":"2023-11-24T06:19:44.617465+0000","last_unstale":"2023-11-24T06:19:46.144529+0000","last_undegraded":"2023-11-24T06:19:46.144529+0000","last_fullsized":"2023-11-24T06:19:46.144529+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:11.876249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624530+0000","last_change":"2023-11-24T06:19:44.617922+0000","last_active":"2023-11-24T06:19:44.624530+0000","last_peered":"2023-11-24T06:19:44.624530+0000","last_clean":"2023-11-24T06:19:44.624530+0000","last_became_active":"2023-11-24T06:19:44.617621+0000","last_became_peered":"2023-11-24T06:19:44.617621+0000","last_unstale":"2023-11-24T06:19:44.624530+0000","last_undegraded":"2023-11-24T06:19:44.624530+0000","last_fullsized":"2023-11-24T06:19:44.624530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:27.706327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.622077+0000","last_change":"2023-11-24T06:19:44.615852+0000","last_active":"2023-11-24T06:19:44.622077+0000","last_peered":"2023-11-24T06:19:44.622077+0000","last_clean":"2023-11-24T06:19:44.622077+0000","last_became_active":"2023-11-24T06:19:44.615557+0000","last_became_peered":"2023-11-24T06:19:44.615557+0000","last_unstale":"2023-11-24T06:19:44.622077+0000","last_undegraded":"2023-11-24T06:19:44.622077+0000","last_fullsized":"2023-11-24T06:19:44.622077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:25.208584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624267+0000","last_change":"2023-11-24T06:19:44.612165+0000","last_active":"2023-11-24T06:19:46.624267+0000","last_peered":"2023-11-24T06:19:46.624267+0000","last_clean":"2023-11-24T06:19:46.624267+0000","last_became_active":"2023-11-24T06:19:44.611888+0000","last_became_peered":"2023-11-24T06:19:44.611888+0000","last_unstale":"2023-11-24T06:19:46.624267+0000","last_undegraded":"2023-11-24T06:19:46.624267+0000","last_fullsized":"2023-11-24T06:19:46.624267+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:21.515369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603370+0000","last_change":"2023-11-24T06:19:44.617293+0000","last_active":"2023-11-24T06:19:45.603370+0000","last_peered":"2023-11-24T06:19:45.603370+0000","last_clean":"2023-11-24T06:19:45.603370+0000","last_became_active":"2023-11-24T06:19:44.616755+0000","last_became_peered":"2023-11-24T06:19:44.616755+0000","last_unstale":"2023-11-24T06:19:45.603370+0000","last_undegraded":"2023-11-24T06:19:45.603370+0000","last_fullsized":"2023-11-24T06:19:45.603370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:11:48.592746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107367+0000","last_change":"2023-11-24T06:19:44.608188+0000","last_active":"2023-11-24T06:19:46.107367+0000","last_peered":"2023-11-24T06:19:46.107367+0000","last_clean":"2023-11-24T06:19:46.107367+0000","last_became_active":"2023-11-24T06:19:44.607822+0000","last_became_peered":"2023-11-24T06:19:44.607822+0000","last_unstale":"2023-11-24T06:19:46.107367+0000","last_undegraded":"2023-11-24T06:19:46.107367+0000","last_fullsized":"2023-11-24T06:19:46.107367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:04:44.567943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605147+0000","last_change":"2023-11-24T06:19:44.609651+0000","last_active":"2023-11-24T06:19:45.605147+0000","last_peered":"2023-11-24T06:19:45.605147+0000","last_clean":"2023-11-24T06:19:45.605147+0000","last_became_active":"2023-11-24T06:19:44.607936+0000","last_became_peered":"2023-11-24T06:19:44.607936+0000","last_unstale":"2023-11-24T06:19:45.605147+0000","last_undegraded":"2023-11-24T06:19:45.605147+0000","last_fullsized":"2023-11-24T06:19:45.605147+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:10:59.958794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107201+0000","last_change":"2023-11-24T06:19:44.616238+0000","last_active":"2023-11-24T06:19:46.107201+0000","last_peered":"2023-11-24T06:19:46.107201+0000","last_clean":"2023-11-24T06:19:46.107201+0000","last_became_active":"2023-11-24T06:19:44.615960+0000","last_became_peered":"2023-11-24T06:19:44.615960+0000","last_unstale":"2023-11-24T06:19:46.107201+0000","last_undegraded":"2023-11-24T06:19:46.107201+0000","last_fullsized":"2023-11-24T06:19:46.107201+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:43.733250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603725+0000","last_change":"2023-11-24T06:19:44.617127+0000","last_active":"2023-11-24T06:19:45.603725+0000","last_peered":"2023-11-24T06:19:45.603725+0000","last_clean":"2023-11-24T06:19:45.603725+0000","last_became_active":"2023-11-24T06:19:44.616734+0000","last_became_peered":"2023-11-24T06:19:44.616734+0000","last_unstale":"2023-11-24T06:19:45.603725+0000","last_undegraded":"2023-11-24T06:19:45.603725+0000","last_fullsized":"2023-11-24T06:19:45.603725+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:58:50.312633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.622466+0000","last_change":"2023-11-24T06:19:44.616104+0000","last_active":"2023-11-24T06:19:44.622466+0000","last_peered":"2023-11-24T06:19:44.622466+0000","last_clean":"2023-11-24T06:19:44.622466+0000","last_became_active":"2023-11-24T06:19:44.615221+0000","last_became_peered":"2023-11-24T06:19:44.615221+0000","last_unstale":"2023-11-24T06:19:44.622466+0000","last_undegraded":"2023-11-24T06:19:44.622466+0000","last_fullsized":"2023-11-24T06:19:44.622466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:23.279015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604143+0000","last_change":"2023-11-24T06:19:44.618110+0000","last_active":"2023-11-24T06:19:45.604143+0000","last_peered":"2023-11-24T06:19:45.604143+0000","last_clean":"2023-11-24T06:19:45.604143+0000","last_became_active":"2023-11-24T06:19:44.617695+0000","last_became_peered":"2023-11-24T06:19:44.617695+0000","last_unstale":"2023-11-24T06:19:45.604143+0000","last_undegraded":"2023-11-24T06:19:45.604143+0000","last_fullsized":"2023-11-24T06:19:45.604143+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:41:17.872131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624885+0000","last_change":"2023-11-24T06:19:44.621432+0000","last_active":"2023-11-24T06:19:44.624885+0000","last_peered":"2023-11-24T06:19:44.624885+0000","last_clean":"2023-11-24T06:19:44.624885+0000","last_became_active":"2023-11-24T06:19:44.618113+0000","last_became_peered":"2023-11-24T06:19:44.618113+0000","last_unstale":"2023-11-24T06:19:44.624885+0000","last_undegraded":"2023-11-24T06:19:44.624885+0000","last_fullsized":"2023-11-24T06:19:44.624885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:57.374633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604110+0000","last_change":"2023-11-24T06:19:44.619350+0000","last_active":"2023-11-24T06:19:45.604110+0000","last_peered":"2023-11-24T06:19:45.604110+0000","last_clean":"2023-11-24T06:19:45.604110+0000","last_became_active":"2023-11-24T06:19:44.618892+0000","last_became_peered":"2023-11-24T06:19:44.618892+0000","last_unstale":"2023-11-24T06:19:45.604110+0000","last_undegraded":"2023-11-24T06:19:45.604110+0000","last_fullsized":"2023-11-24T06:19:45.604110+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:17.149210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624461+0000","last_change":"2023-11-24T06:19:44.612782+0000","last_active":"2023-11-24T06:19:46.624461+0000","last_peered":"2023-11-24T06:19:46.624461+0000","last_clean":"2023-11-24T06:19:46.624461+0000","last_became_active":"2023-11-24T06:19:44.612491+0000","last_became_peered":"2023-11-24T06:19:44.612491+0000","last_unstale":"2023-11-24T06:19:46.624461+0000","last_undegraded":"2023-11-24T06:19:46.624461+0000","last_fullsized":"2023-11-24T06:19:46.624461+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:07:41.777438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.993094+0000","last_change":"2023-11-24T06:19:43.603470+0000","last_active":"2023-11-24T06:19:44.993094+0000","last_peered":"2023-11-24T06:19:44.993094+0000","last_clean":"2023-11-24T06:19:44.993094+0000","last_became_active":"2023-11-24T06:19:43.603222+0000","last_became_peered":"2023-11-24T06:19:43.603222+0000","last_unstale":"2023-11-24T06:19:44.993094+0000","last_undegraded":"2023-11-24T06:19:44.993094+0000","last_fullsized":"2023-11-24T06:19:44.993094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:05.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.627205+0000","last_change":"2023-11-24T06:19:44.622557+0000","last_active":"2023-11-24T06:19:44.627205+0000","last_peered":"2023-11-24T06:19:44.627205+0000","last_clean":"2023-11-24T06:19:44.627205+0000","last_became_active":"2023-11-24T06:19:44.622258+0000","last_became_peered":"2023-11-24T06:19:44.622258+0000","last_unstale":"2023-11-24T06:19:44.627205+0000","last_undegraded":"2023-11-24T06:19:44.627205+0000","last_fullsized":"2023-11-24T06:19:44.627205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:11.876249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624434+0000","last_change":"2023-11-24T06:19:43.601904+0000","last_active":"2023-11-24T06:19:46.624434+0000","last_peered":"2023-11-24T06:19:46.624434+0000","last_clean":"2023-11-24T06:19:46.624434+0000","last_became_active":"2023-11-24T06:19:43.601620+0000","last_became_peered":"2023-11-24T06:19:43.601620+0000","last_unstale":"2023-11-24T06:19:46.624434+0000","last_undegraded":"2023-11-24T06:19:46.624434+0000","last_fullsized":"2023-11-24T06:19:46.624434+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:05.997314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624414+0000","last_change":"2023-11-24T06:19:44.616895+0000","last_active":"2023-11-24T06:19:44.624414+0000","last_peered":"2023-11-24T06:19:44.624414+0000","last_clean":"2023-11-24T06:19:44.624414+0000","last_became_active":"2023-11-24T06:19:44.616610+0000","last_became_peered":"2023-11-24T06:19:44.616610+0000","last_unstale":"2023-11-24T06:19:44.624414+0000","last_undegraded":"2023-11-24T06:19:44.624414+0000","last_fullsized":"2023-11-24T06:19:44.624414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:01.710358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604745+0000","last_change":"2023-11-24T06:19:43.607012+0000","last_active":"2023-11-24T06:19:45.604745+0000","last_peered":"2023-11-24T06:19:45.604745+0000","last_clean":"2023-11-24T06:19:45.604745+0000","last_became_active":"2023-11-24T06:19:43.606729+0000","last_became_peered":"2023-11-24T06:19:43.606729+0000","last_unstale":"2023-11-24T06:19:45.604745+0000","last_undegraded":"2023-11-24T06:19:45.604745+0000","last_fullsized":"2023-11-24T06:19:45.604745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:11:47.584570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107212+0000","last_change":"2023-11-24T06:19:44.616126+0000","last_active":"2023-11-24T06:19:46.107212+0000","last_peered":"2023-11-24T06:19:46.107212+0000","last_clean":"2023-11-24T06:19:46.107212+0000","last_became_active":"2023-11-24T06:19:44.615730+0000","last_became_peered":"2023-11-24T06:19:44.615730+0000","last_unstale":"2023-11-24T06:19:46.107212+0000","last_undegraded":"2023-11-24T06:19:46.107212+0000","last_fullsized":"2023-11-24T06:19:46.107212+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:17.543360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144559+0000","last_change":"2023-11-24T06:19:43.608044+0000","last_active":"2023-11-24T06:19:46.144559+0000","last_peered":"2023-11-24T06:19:46.144559+0000","last_clean":"2023-11-24T06:19:46.144559+0000","last_became_active":"2023-11-24T06:19:43.607808+0000","last_became_peered":"2023-11-24T06:19:43.607808+0000","last_unstale":"2023-11-24T06:19:46.144559+0000","last_undegraded":"2023-11-24T06:19:46.144559+0000","last_fullsized":"2023-11-24T06:19:46.144559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:26.698151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603696+0000","last_change":"2023-11-24T06:19:44.617935+0000","last_active":"2023-11-24T06:19:45.603696+0000","last_peered":"2023-11-24T06:19:45.603696+0000","last_clean":"2023-11-24T06:19:45.603696+0000","last_became_active":"2023-11-24T06:19:44.617352+0000","last_became_peered":"2023-11-24T06:19:44.617352+0000","last_unstale":"2023-11-24T06:19:45.603696+0000","last_undegraded":"2023-11-24T06:19:45.603696+0000","last_fullsized":"2023-11-24T06:19:45.603696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:11:48.592746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.992772+0000","last_change":"2023-11-24T06:19:43.599161+0000","last_active":"2023-11-24T06:19:44.992772+0000","last_peered":"2023-11-24T06:19:44.992772+0000","last_clean":"2023-11-24T06:19:44.992772+0000","last_became_active":"2023-11-24T06:19:43.598849+0000","last_became_peered":"2023-11-24T06:19:43.598849+0000","last_unstale":"2023-11-24T06:19:44.992772+0000","last_undegraded":"2023-11-24T06:19:44.992772+0000","last_fullsized":"2023-11-24T06:19:44.992772+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:05.997314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603199+0000","last_change":"2023-11-24T06:19:44.619679+0000","last_active":"2023-11-24T06:19:45.603199+0000","last_peered":"2023-11-24T06:19:45.603199+0000","last_clean":"2023-11-24T06:19:45.603199+0000","last_became_active":"2023-11-24T06:19:44.619280+0000","last_became_peered":"2023-11-24T06:19:44.619280+0000","last_unstale":"2023-11-24T06:19:45.603199+0000","last_undegraded":"2023-11-24T06:19:45.603199+0000","last_fullsized":"2023-11-24T06:19:45.603199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:43.733250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.992818+0000","last_change":"2023-11-24T06:19:43.599222+0000","last_active":"2023-11-24T06:19:44.992818+0000","last_peered":"2023-11-24T06:19:44.992818+0000","last_clean":"2023-11-24T06:19:44.992818+0000","last_became_active":"2023-11-24T06:19:43.598908+0000","last_became_peered":"2023-11-24T06:19:43.598908+0000","last_unstale":"2023-11-24T06:19:44.992818+0000","last_undegraded":"2023-11-24T06:19:44.992818+0000","last_fullsized":"2023-11-24T06:19:44.992818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:12.352766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624687+0000","last_change":"2023-11-24T06:19:44.618829+0000","last_active":"2023-11-24T06:19:44.624687+0000","last_peered":"2023-11-24T06:19:44.624687+0000","last_clean":"2023-11-24T06:19:44.624687+0000","last_became_active":"2023-11-24T06:19:44.618507+0000","last_became_peered":"2023-11-24T06:19:44.618507+0000","last_unstale":"2023-11-24T06:19:44.624687+0000","last_undegraded":"2023-11-24T06:19:44.624687+0000","last_fullsized":"2023-11-24T06:19:44.624687+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:03.428940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604963+0000","last_change":"2023-11-24T06:19:43.605710+0000","last_active":"2023-11-24T06:19:45.604963+0000","last_peered":"2023-11-24T06:19:45.604963+0000","last_clean":"2023-11-24T06:19:45.604963+0000","last_became_active":"2023-11-24T06:19:43.605085+0000","last_became_peered":"2023-11-24T06:19:43.605085+0000","last_unstale":"2023-11-24T06:19:45.604963+0000","last_undegraded":"2023-11-24T06:19:45.604963+0000","last_fullsized":"2023-11-24T06:19:45.604963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.994336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106814+0000","last_change":"2023-11-24T06:19:44.618009+0000","last_active":"2023-11-24T06:19:46.106814+0000","last_peered":"2023-11-24T06:19:46.106814+0000","last_clean":"2023-11-24T06:19:46.106814+0000","last_became_active":"2023-11-24T06:19:44.617747+0000","last_became_peered":"2023-11-24T06:19:44.617747+0000","last_unstale":"2023-11-24T06:19:46.106814+0000","last_undegraded":"2023-11-24T06:19:46.106814+0000","last_fullsized":"2023-11-24T06:19:46.106814+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:51:22.442670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.609487+0000","last_change":"2023-11-24T06:19:43.594541+0000","last_active":"2023-11-24T06:19:44.609487+0000","last_peered":"2023-11-24T06:19:44.609487+0000","last_clean":"2023-11-24T06:19:44.609487+0000","last_became_active":"2023-11-24T06:19:43.594258+0000","last_became_peered":"2023-11-24T06:19:43.594258+0000","last_unstale":"2023-11-24T06:19:44.609487+0000","last_undegraded":"2023-11-24T06:19:44.609487+0000","last_fullsized":"2023-11-24T06:19:44.609487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:10.868073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603922+0000","last_change":"2023-11-24T06:19:44.620214+0000","last_active":"2023-11-24T06:19:45.603922+0000","last_peered":"2023-11-24T06:19:45.603922+0000","last_clean":"2023-11-24T06:19:45.603922+0000","last_became_active":"2023-11-24T06:19:44.619768+0000","last_became_peered":"2023-11-24T06:19:44.619768+0000","last_unstale":"2023-11-24T06:19:45.603922+0000","last_undegraded":"2023-11-24T06:19:45.603922+0000","last_fullsized":"2023-11-24T06:19:45.603922+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:35:04.205238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604887+0000","last_change":"2023-11-24T06:19:43.603672+0000","last_active":"2023-11-24T06:19:45.604887+0000","last_peered":"2023-11-24T06:19:45.604887+0000","last_clean":"2023-11-24T06:19:45.604887+0000","last_became_active":"2023-11-24T06:19:43.602527+0000","last_became_peered":"2023-11-24T06:19:43.602527+0000","last_unstale":"2023-11-24T06:19:45.604887+0000","last_undegraded":"2023-11-24T06:19:45.604887+0000","last_fullsized":"2023-11-24T06:19:45.604887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:04:43.559768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605063+0000","last_change":"2023-11-24T06:19:44.609926+0000","last_active":"2023-11-24T06:19:45.605063+0000","last_peered":"2023-11-24T06:19:45.605063+0000","last_clean":"2023-11-24T06:19:45.605063+0000","last_became_active":"2023-11-24T06:19:44.609653+0000","last_became_peered":"2023-11-24T06:19:44.609653+0000","last_unstale":"2023-11-24T06:19:45.605063+0000","last_undegraded":"2023-11-24T06:19:45.605063+0000","last_fullsized":"2023-11-24T06:19:45.605063+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:53:46.166270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624511+0000","last_change":"2023-11-24T06:19:43.606931+0000","last_active":"2023-11-24T06:19:46.624511+0000","last_peered":"2023-11-24T06:19:46.624511+0000","last_clean":"2023-11-24T06:19:46.624511+0000","last_became_active":"2023-11-24T06:19:43.606619+0000","last_became_peered":"2023-11-24T06:19:43.606619+0000","last_unstale":"2023-11-24T06:19:46.624511+0000","last_undegraded":"2023-11-24T06:19:46.624511+0000","last_fullsized":"2023-11-24T06:19:46.624511+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:56.366458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604855+0000","last_change":"2023-11-24T06:19:44.605015+0000","last_active":"2023-11-24T06:19:45.604855+0000","last_peered":"2023-11-24T06:19:45.604855+0000","last_clean":"2023-11-24T06:19:45.604855+0000","last_became_active":"2023-11-24T06:19:44.604692+0000","last_became_peered":"2023-11-24T06:19:44.604692+0000","last_unstale":"2023-11-24T06:19:45.604855+0000","last_undegraded":"2023-11-24T06:19:45.604855+0000","last_fullsized":"2023-11-24T06:19:45.604855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:54.283680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603980+0000","last_change":"2023-11-24T06:19:43.602959+0000","last_active":"2023-11-24T06:19:45.603980+0000","last_peered":"2023-11-24T06:19:45.603980+0000","last_clean":"2023-11-24T06:19:45.603980+0000","last_became_active":"2023-11-24T06:19:43.602599+0000","last_became_peered":"2023-11-24T06:19:43.602599+0000","last_unstale":"2023-11-24T06:19:45.603980+0000","last_undegraded":"2023-11-24T06:19:45.603980+0000","last_fullsized":"2023-11-24T06:19:45.603980+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:55:10.412916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603873+0000","last_change":"2023-11-24T06:19:44.620784+0000","last_active":"2023-11-24T06:19:45.603873+0000","last_peered":"2023-11-24T06:19:45.603873+0000","last_clean":"2023-11-24T06:19:45.603873+0000","last_became_active":"2023-11-24T06:19:44.620154+0000","last_became_peered":"2023-11-24T06:19:44.620154+0000","last_unstale":"2023-11-24T06:19:45.603873+0000","last_undegraded":"2023-11-24T06:19:45.603873+0000","last_fullsized":"2023-11-24T06:19:45.603873+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:24.264848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605006+0000","last_change":"2023-11-24T06:19:43.607375+0000","last_active":"2023-11-24T06:19:45.605006+0000","last_peered":"2023-11-24T06:19:45.605006+0000","last_clean":"2023-11-24T06:19:45.605006+0000","last_became_active":"2023-11-24T06:19:43.607119+0000","last_became_peered":"2023-11-24T06:19:43.607119+0000","last_unstale":"2023-11-24T06:19:45.605006+0000","last_undegraded":"2023-11-24T06:19:45.605006+0000","last_fullsized":"2023-11-24T06:19:45.605006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:16.535184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106880+0000","last_change":"2023-11-24T06:19:44.618619+0000","last_active":"2023-11-24T06:19:46.106880+0000","last_peered":"2023-11-24T06:19:46.106880+0000","last_clean":"2023-11-24T06:19:46.106880+0000","last_became_active":"2023-11-24T06:19:44.618344+0000","last_became_peered":"2023-11-24T06:19:44.618344+0000","last_unstale":"2023-11-24T06:19:46.106880+0000","last_undegraded":"2023-11-24T06:19:46.106880+0000","last_fullsized":"2023-11-24T06:19:46.106880+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:55:11.421092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.991215+0000","last_change":"2023-11-24T06:19:43.597304+0000","last_active":"2023-11-24T06:19:44.991215+0000","last_peered":"2023-11-24T06:19:44.991215+0000","last_clean":"2023-11-24T06:19:44.991215+0000","last_became_active":"2023-11-24T06:19:43.597030+0000","last_became_peered":"2023-11-24T06:19:43.597030+0000","last_unstale":"2023-11-24T06:19:44.991215+0000","last_undegraded":"2023-11-24T06:19:44.991215+0000","last_fullsized":"2023-11-24T06:19:44.991215+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:36:45.636780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605142+0000","last_change":"2023-11-24T06:19:44.615720+0000","last_active":"2023-11-24T06:19:45.605142+0000","last_peered":"2023-11-24T06:19:45.605142+0000","last_clean":"2023-11-24T06:19:45.605142+0000","last_became_active":"2023-11-24T06:19:44.615442+0000","last_became_peered":"2023-11-24T06:19:44.615442+0000","last_unstale":"2023-11-24T06:19:45.605142+0000","last_undegraded":"2023-11-24T06:19:45.605142+0000","last_fullsized":"2023-11-24T06:19:45.605142+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:58.700838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106842+0000","last_change":"2023-11-24T06:19:43.602015+0000","last_active":"2023-11-24T06:19:46.106842+0000","last_peered":"2023-11-24T06:19:46.106842+0000","last_clean":"2023-11-24T06:19:46.106842+0000","last_became_active":"2023-11-24T06:19:43.601730+0000","last_became_peered":"2023-11-24T06:19:43.601730+0000","last_unstale":"2023-11-24T06:19:46.106842+0000","last_undegraded":"2023-11-24T06:19:46.106842+0000","last_fullsized":"2023-11-24T06:19:46.106842+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:10.868073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604131+0000","last_change":"2023-11-24T06:19:44.619158+0000","last_active":"2023-11-24T06:19:45.604131+0000","last_peered":"2023-11-24T06:19:45.604131+0000","last_clean":"2023-11-24T06:19:45.604131+0000","last_became_active":"2023-11-24T06:19:44.618560+0000","last_became_peered":"2023-11-24T06:19:44.618560+0000","last_unstale":"2023-11-24T06:19:45.604131+0000","last_undegraded":"2023-11-24T06:19:45.604131+0000","last_fullsized":"2023-11-24T06:19:45.604131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:34.436870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604710+0000","last_change":"2023-11-24T06:19:43.603182+0000","last_active":"2023-11-24T06:19:45.604710+0000","last_peered":"2023-11-24T06:19:45.604710+0000","last_clean":"2023-11-24T06:19:45.604710+0000","last_became_active":"2023-11-24T06:19:43.602910+0000","last_became_peered":"2023-11-24T06:19:43.602910+0000","last_unstale":"2023-11-24T06:19:45.604710+0000","last_undegraded":"2023-11-24T06:19:45.604710+0000","last_fullsized":"2023-11-24T06:19:45.604710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:42.725075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.627267+0000","last_change":"2023-11-24T06:19:44.623347+0000","last_active":"2023-11-24T06:19:44.627267+0000","last_peered":"2023-11-24T06:19:44.627267+0000","last_clean":"2023-11-24T06:19:44.627267+0000","last_became_active":"2023-11-24T06:19:44.623005+0000","last_became_peered":"2023-11-24T06:19:44.623005+0000","last_unstale":"2023-11-24T06:19:44.627267+0000","last_undegraded":"2023-11-24T06:19:44.627267+0000","last_fullsized":"2023-11-24T06:19:44.627267+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:25:57.248982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604431+0000","last_change":"2023-11-24T06:19:43.606229+0000","last_active":"2023-11-24T06:19:45.604431+0000","last_peered":"2023-11-24T06:19:45.604431+0000","last_clean":"2023-11-24T06:19:45.604431+0000","last_became_active":"2023-11-24T06:19:43.605970+0000","last_became_peered":"2023-11-24T06:19:43.605970+0000","last_unstale":"2023-11-24T06:19:45.604431+0000","last_undegraded":"2023-11-24T06:19:45.604431+0000","last_fullsized":"2023-11-24T06:19:45.604431+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:21.570607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624987+0000","last_change":"2023-11-24T06:19:44.623354+0000","last_active":"2023-11-24T06:19:44.624987+0000","last_peered":"2023-11-24T06:19:44.624987+0000","last_clean":"2023-11-24T06:19:44.624987+0000","last_became_active":"2023-11-24T06:19:44.623070+0000","last_became_peered":"2023-11-24T06:19:44.623070+0000","last_unstale":"2023-11-24T06:19:44.624987+0000","last_undegraded":"2023-11-24T06:19:44.624987+0000","last_fullsized":"2023-11-24T06:19:44.624987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:25:57.248982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107619+0000","last_change":"2023-11-24T06:19:43.609838+0000","last_active":"2023-11-24T06:19:46.107619+0000","last_peered":"2023-11-24T06:19:46.107619+0000","last_clean":"2023-11-24T06:19:46.107619+0000","last_became_active":"2023-11-24T06:19:43.609553+0000","last_became_peered":"2023-11-24T06:19:43.609553+0000","last_unstale":"2023-11-24T06:19:46.107619+0000","last_undegraded":"2023-11-24T06:19:46.107619+0000","last_fullsized":"2023-11-24T06:19:46.107619+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:07:40.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604218+0000","last_change":"2023-11-24T06:19:44.623125+0000","last_active":"2023-11-24T06:19:45.604218+0000","last_peered":"2023-11-24T06:19:45.604218+0000","last_clean":"2023-11-24T06:19:45.604218+0000","last_became_active":"2023-11-24T06:19:44.622866+0000","last_became_peered":"2023-11-24T06:19:44.622866+0000","last_unstale":"2023-11-24T06:19:45.604218+0000","last_undegraded":"2023-11-24T06:19:45.604218+0000","last_fullsized":"2023-11-24T06:19:45.604218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:17.149210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.991123+0000","last_change":"2023-11-24T06:19:43.601747+0000","last_active":"2023-11-24T06:19:44.991123+0000","last_peered":"2023-11-24T06:19:44.991123+0000","last_clean":"2023-11-24T06:19:44.991123+0000","last_became_active":"2023-11-24T06:19:43.601458+0000","last_became_peered":"2023-11-24T06:19:43.601458+0000","last_unstale":"2023-11-24T06:19:44.991123+0000","last_undegraded":"2023-11-24T06:19:44.991123+0000","last_fullsized":"2023-11-24T06:19:44.991123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:56.366458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107052+0000","last_change":"2023-11-24T06:19:44.620224+0000","last_active":"2023-11-24T06:19:46.107052+0000","last_peered":"2023-11-24T06:19:46.107052+0000","last_clean":"2023-11-24T06:19:46.107052+0000","last_became_active":"2023-11-24T06:19:44.619946+0000","last_became_peered":"2023-11-24T06:19:44.619946+0000","last_unstale":"2023-11-24T06:19:46.107052+0000","last_undegraded":"2023-11-24T06:19:46.107052+0000","last_fullsized":"2023-11-24T06:19:46.107052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:34.002512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.994984+0000","last_change":"2023-11-24T06:19:43.605040+0000","last_active":"2023-11-24T06:19:44.994984+0000","last_peered":"2023-11-24T06:19:44.994984+0000","last_clean":"2023-11-24T06:19:44.994984+0000","last_became_active":"2023-11-24T06:19:43.604815+0000","last_became_peered":"2023-11-24T06:19:43.604815+0000","last_unstale":"2023-11-24T06:19:44.994984+0000","last_undegraded":"2023-11-24T06:19:44.994984+0000","last_fullsized":"2023-11-24T06:19:44.994984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:59.804014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144699+0000","last_change":"2023-11-24T06:19:44.605133+0000","last_active":"2023-11-24T06:19:46.144699+0000","last_peered":"2023-11-24T06:19:46.144699+0000","last_clean":"2023-11-24T06:19:46.144699+0000","last_became_active":"2023-11-24T06:19:44.604725+0000","last_became_peered":"2023-11-24T06:19:44.604725+0000","last_unstale":"2023-11-24T06:19:46.144699+0000","last_undegraded":"2023-11-24T06:19:46.144699+0000","last_fullsized":"2023-11-24T06:19:46.144699+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:06.278159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.610050+0000","last_change":"2023-11-24T06:19:43.599023+0000","last_active":"2023-11-24T06:19:44.610050+0000","last_peered":"2023-11-24T06:19:44.610050+0000","last_clean":"2023-11-24T06:19:44.610050+0000","last_became_active":"2023-11-24T06:19:43.598728+0000","last_became_peered":"2023-11-24T06:19:43.598728+0000","last_unstale":"2023-11-24T06:19:44.610050+0000","last_undegraded":"2023-11-24T06:19:44.610050+0000","last_fullsized":"2023-11-24T06:19:44.610050+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:30.429443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144510+0000","last_change":"2023-11-24T06:19:44.620299+0000","last_active":"2023-11-24T06:19:46.144510+0000","last_peered":"2023-11-24T06:19:46.144510+0000","last_clean":"2023-11-24T06:19:46.144510+0000","last_became_active":"2023-11-24T06:19:44.619966+0000","last_became_peered":"2023-11-24T06:19:44.619966+0000","last_unstale":"2023-11-24T06:19:46.144510+0000","last_undegraded":"2023-11-24T06:19:46.144510+0000","last_fullsized":"2023-11-24T06:19:46.144510+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:25:57.248982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624436+0000","last_change":"2023-11-24T06:19:43.607279+0000","last_active":"2023-11-24T06:19:46.624436+0000","last_peered":"2023-11-24T06:19:46.624436+0000","last_clean":"2023-11-24T06:19:46.624436+0000","last_became_active":"2023-11-24T06:19:43.607024+0000","last_became_peered":"2023-11-24T06:19:43.607024+0000","last_unstale":"2023-11-24T06:19:46.624436+0000","last_undegraded":"2023-11-24T06:19:46.624436+0000","last_fullsized":"2023-11-24T06:19:46.624436+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:30.429443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144583+0000","last_change":"2023-11-24T06:19:44.615777+0000","last_active":"2023-11-24T06:19:46.144583+0000","last_peered":"2023-11-24T06:19:46.144583+0000","last_clean":"2023-11-24T06:19:46.144583+0000","last_became_active":"2023-11-24T06:19:44.615495+0000","last_became_peered":"2023-11-24T06:19:44.615495+0000","last_unstale":"2023-11-24T06:19:46.144583+0000","last_undegraded":"2023-11-24T06:19:46.144583+0000","last_fullsized":"2023-11-24T06:19:46.144583+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:07.005490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.610512+0000","last_change":"2023-11-24T06:19:43.606633+0000","last_active":"2023-11-24T06:19:44.610512+0000","last_peered":"2023-11-24T06:19:44.610512+0000","last_clean":"2023-11-24T06:19:44.610512+0000","last_became_active":"2023-11-24T06:19:43.606347+0000","last_became_peered":"2023-11-24T06:19:43.606347+0000","last_unstale":"2023-11-24T06:19:44.610512+0000","last_undegraded":"2023-11-24T06:19:44.610512+0000","last_fullsized":"2023-11-24T06:19:44.610512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:54.775345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144487+0000","last_change":"2023-11-24T06:19:44.615445+0000","last_active":"2023-11-24T06:19:46.144487+0000","last_peered":"2023-11-24T06:19:46.144487+0000","last_clean":"2023-11-24T06:19:46.144487+0000","last_became_active":"2023-11-24T06:19:44.615034+0000","last_became_peered":"2023-11-24T06:19:44.615034+0000","last_unstale":"2023-11-24T06:19:46.144487+0000","last_undegraded":"2023-11-24T06:19:46.144487+0000","last_fullsized":"2023-11-24T06:19:46.144487+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:01.710358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.609592+0000","last_change":"2023-11-24T06:19:43.606153+0000","last_active":"2023-11-24T06:19:44.609592+0000","last_peered":"2023-11-24T06:19:44.609592+0000","last_clean":"2023-11-24T06:19:44.609592+0000","last_became_active":"2023-11-24T06:19:43.605891+0000","last_became_peered":"2023-11-24T06:19:43.605891+0000","last_unstale":"2023-11-24T06:19:44.609592+0000","last_undegraded":"2023-11-24T06:19:44.609592+0000","last_fullsized":"2023-11-24T06:19:44.609592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:54.775345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603558+0000","last_change":"2023-11-24T06:19:44.615573+0000","last_active":"2023-11-24T06:19:45.603558+0000","last_peered":"2023-11-24T06:19:45.603558+0000","last_clean":"2023-11-24T06:19:45.603558+0000","last_became_active":"2023-11-24T06:19:44.615282+0000","last_became_peered":"2023-11-24T06:19:44.615282+0000","last_unstale":"2023-11-24T06:19:45.603558+0000","last_undegraded":"2023-11-24T06:19:45.603558+0000","last_fullsized":"2023-11-24T06:19:45.603558+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:17.543360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603030+0000","last_change":"2023-11-24T06:19:43.602857+0000","last_active":"2023-11-24T06:19:45.603030+0000","last_peered":"2023-11-24T06:19:45.603030+0000","last_clean":"2023-11-24T06:19:45.603030+0000","last_became_active":"2023-11-24T06:19:43.602545+0000","last_became_peered":"2023-11-24T06:19:43.602545+0000","last_unstale":"2023-11-24T06:19:45.603030+0000","last_undegraded":"2023-11-24T06:19:45.603030+0000","last_fullsized":"2023-11-24T06:19:45.603030+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:30.429443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144586+0000","last_change":"2023-11-24T06:19:44.623291+0000","last_active":"2023-11-24T06:19:46.144586+0000","last_peered":"2023-11-24T06:19:46.144586+0000","last_clean":"2023-11-24T06:19:46.144586+0000","last_became_active":"2023-11-24T06:19:44.622972+0000","last_became_peered":"2023-11-24T06:19:44.622972+0000","last_unstale":"2023-11-24T06:19:46.144586+0000","last_undegraded":"2023-11-24T06:19:46.144586+0000","last_fullsized":"2023-11-24T06:19:46.144586+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:55.783521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107458+0000","last_change":"2023-11-24T06:19:43.608298+0000","last_active":"2023-11-24T06:19:46.107458+0000","last_peered":"2023-11-24T06:19:46.107458+0000","last_clean":"2023-11-24T06:19:46.107458+0000","last_became_active":"2023-11-24T06:19:43.606141+0000","last_became_peered":"2023-11-24T06:19:43.606141+0000","last_unstale":"2023-11-24T06:19:46.107458+0000","last_undegraded":"2023-11-24T06:19:46.107458+0000","last_fullsized":"2023-11-24T06:19:46.107458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:36:45.636780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604155+0000","last_change":"2023-11-24T06:19:44.618409+0000","last_active":"2023-11-24T06:19:45.604155+0000","last_peered":"2023-11-24T06:19:45.604155+0000","last_clean":"2023-11-24T06:19:45.604155+0000","last_became_active":"2023-11-24T06:19:44.618085+0000","last_became_peered":"2023-11-24T06:19:44.618085+0000","last_unstale":"2023-11-24T06:19:45.604155+0000","last_undegraded":"2023-11-24T06:19:45.604155+0000","last_fullsized":"2023-11-24T06:19:45.604155+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:34.436870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.625760+0000","last_change":"2023-11-24T06:19:43.606647+0000","last_active":"2023-11-24T06:19:46.625760+0000","last_peered":"2023-11-24T06:19:46.625760+0000","last_clean":"2023-11-24T06:19:46.625760+0000","last_became_active":"2023-11-24T06:19:43.606298+0000","last_became_peered":"2023-11-24T06:19:43.606298+0000","last_unstale":"2023-11-24T06:19:46.625760+0000","last_undegraded":"2023-11-24T06:19:46.625760+0000","last_fullsized":"2023-11-24T06:19:46.625760+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:05.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624350+0000","last_change":"2023-11-24T06:19:44.612790+0000","last_active":"2023-11-24T06:19:46.624350+0000","last_peered":"2023-11-24T06:19:46.624350+0000","last_clean":"2023-11-24T06:19:46.624350+0000","last_became_active":"2023-11-24T06:19:44.612566+0000","last_became_peered":"2023-11-24T06:19:44.612566+0000","last_unstale":"2023-11-24T06:19:46.624350+0000","last_undegraded":"2023-11-24T06:19:46.624350+0000","last_fullsized":"2023-11-24T06:19:46.624350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:28:29.540648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107102+0000","last_change":"2023-11-24T06:19:43.606597+0000","last_active":"2023-11-24T06:19:46.107102+0000","last_peered":"2023-11-24T06:19:46.107102+0000","last_clean":"2023-11-24T06:19:46.107102+0000","last_became_active":"2023-11-24T06:19:43.606174+0000","last_became_peered":"2023-11-24T06:19:43.606174+0000","last_unstale":"2023-11-24T06:19:46.107102+0000","last_undegraded":"2023-11-24T06:19:46.107102+0000","last_fullsized":"2023-11-24T06:19:46.107102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:59.804014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.622556+0000","last_change":"2023-11-24T06:19:44.617331+0000","last_active":"2023-11-24T06:19:44.622556+0000","last_peered":"2023-11-24T06:19:44.622556+0000","last_clean":"2023-11-24T06:19:44.622556+0000","last_became_active":"2023-11-24T06:19:44.616977+0000","last_became_peered":"2023-11-24T06:19:44.616977+0000","last_unstale":"2023-11-24T06:19:44.622556+0000","last_undegraded":"2023-11-24T06:19:44.622556+0000","last_fullsized":"2023-11-24T06:19:44.622556+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:03.428940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624390+0000","last_change":"2023-11-24T06:19:43.603879+0000","last_active":"2023-11-24T06:19:46.624390+0000","last_peered":"2023-11-24T06:19:46.624390+0000","last_clean":"2023-11-24T06:19:46.624390+0000","last_became_active":"2023-11-24T06:19:43.603611+0000","last_became_peered":"2023-11-24T06:19:43.603611+0000","last_unstale":"2023-11-24T06:19:46.624390+0000","last_undegraded":"2023-11-24T06:19:46.624390+0000","last_fullsized":"2023-11-24T06:19:46.624390+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:59.804014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624492+0000","last_change":"2023-11-24T06:19:44.618178+0000","last_active":"2023-11-24T06:19:46.624492+0000","last_peered":"2023-11-24T06:19:46.624492+0000","last_clean":"2023-11-24T06:19:46.624492+0000","last_became_active":"2023-11-24T06:19:44.617903+0000","last_became_peered":"2023-11-24T06:19:44.617903+0000","last_unstale":"2023-11-24T06:19:46.624492+0000","last_undegraded":"2023-11-24T06:19:46.624492+0000","last_fullsized":"2023-11-24T06:19:46.624492+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:45:21.055969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106926+0000","last_change":"2023-11-24T06:19:43.604070+0000","last_active":"2023-11-24T06:19:46.106926+0000","last_peered":"2023-11-24T06:19:46.106926+0000","last_clean":"2023-11-24T06:19:46.106926+0000","last_became_active":"2023-11-24T06:19:43.603750+0000","last_became_peered":"2023-11-24T06:19:43.603750+0000","last_unstale":"2023-11-24T06:19:46.106926+0000","last_undegraded":"2023-11-24T06:19:46.106926+0000","last_fullsized":"2023-11-24T06:19:46.106926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:56.366458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604484+0000","last_change":"2023-11-24T06:19:44.609460+0000","last_active":"2023-11-24T06:19:45.604484+0000","last_peered":"2023-11-24T06:19:45.604484+0000","last_clean":"2023-11-24T06:19:45.604484+0000","last_became_active":"2023-11-24T06:19:44.609125+0000","last_became_peered":"2023-11-24T06:19:44.609125+0000","last_unstale":"2023-11-24T06:19:45.604484+0000","last_undegraded":"2023-11-24T06:19:45.604484+0000","last_fullsized":"2023-11-24T06:19:45.604484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:35:04.205238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.609699+0000","last_change":"2023-11-24T06:19:43.603124+0000","last_active":"2023-11-24T06:19:44.609699+0000","last_peered":"2023-11-24T06:19:44.609699+0000","last_clean":"2023-11-24T06:19:44.609699+0000","last_became_active":"2023-11-24T06:19:43.602856+0000","last_became_peered":"2023-11-24T06:19:43.602856+0000","last_unstale":"2023-11-24T06:19:44.609699+0000","last_undegraded":"2023-11-24T06:19:44.609699+0000","last_fullsized":"2023-11-24T06:19:44.609699+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:05.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.622114+0000","last_change":"2023-11-24T06:19:44.615975+0000","last_active":"2023-11-24T06:19:44.622114+0000","last_peered":"2023-11-24T06:19:44.622114+0000","last_clean":"2023-11-24T06:19:44.622114+0000","last_became_active":"2023-11-24T06:19:44.615687+0000","last_became_peered":"2023-11-24T06:19:44.615687+0000","last_unstale":"2023-11-24T06:19:44.622114+0000","last_undegraded":"2023-11-24T06:19:44.622114+0000","last_fullsized":"2023-11-24T06:19:44.622114+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:45.302016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605071+0000","last_change":"2023-11-24T06:19:43.603830+0000","last_active":"2023-11-24T06:19:45.605071+0000","last_peered":"2023-11-24T06:19:45.605071+0000","last_clean":"2023-11-24T06:19:45.605071+0000","last_became_active":"2023-11-24T06:19:43.602570+0000","last_became_peered":"2023-11-24T06:19:43.602570+0000","last_unstale":"2023-11-24T06:19:45.605071+0000","last_undegraded":"2023-11-24T06:19:45.605071+0000","last_fullsized":"2023-11-24T06:19:45.605071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:58:49.304457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603934+0000","last_change":"2023-11-24T06:19:44.606372+0000","last_active":"2023-11-24T06:19:45.603934+0000","last_peered":"2023-11-24T06:19:45.603934+0000","last_clean":"2023-11-24T06:19:45.603934+0000","last_became_active":"2023-11-24T06:19:44.606069+0000","last_became_peered":"2023-11-24T06:19:44.606069+0000","last_unstale":"2023-11-24T06:19:45.603934+0000","last_undegraded":"2023-11-24T06:19:45.603934+0000","last_fullsized":"2023-11-24T06:19:45.603934+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:58:50.312633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.609826+0000","last_change":"2023-11-24T06:19:43.607414+0000","last_active":"2023-11-24T06:19:44.609826+0000","last_peered":"2023-11-24T06:19:44.609826+0000","last_clean":"2023-11-24T06:19:44.609826+0000","last_became_active":"2023-11-24T06:19:43.607080+0000","last_became_peered":"2023-11-24T06:19:43.607080+0000","last_unstale":"2023-11-24T06:19:44.609826+0000","last_undegraded":"2023-11-24T06:19:44.609826+0000","last_fullsized":"2023-11-24T06:19:44.609826+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:00.702182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.600597+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:19:44.600597+0000","last_peered":"2023-11-24T06:19:44.600597+0000","last_clean":"2023-11-24T06:19:44.600597+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:19:44.600597+0000","last_undegraded":"2023-11-24T06:19:44.600597+0000","last_fullsized":"2023-11-24T06:19:44.600597+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":741583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.64600000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.53600000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.502}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.61699999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.58399999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.67500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":145441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.59399999999999997}]},{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":6.423,"5min":6.423,"15min":6.423},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.44400000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":12.554,"5min":12.554,"15min":12.554},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.65800000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.621}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":152759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.495}]},{"osd":3,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.62},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.59799999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.746}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":735193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.26600000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.31},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.34300000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.433}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.54100000000000004}]},{"osd":6,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.40200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":151195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.23899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.372}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.32400000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.35299999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.44800000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.39200000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":152759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.246},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.378}]},{"osd":1,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.31},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.36299999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.435}]},{"osd":7,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":144929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.496},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.64900000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.63100000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.374},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.40600000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59399999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":151797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.04,"5min":3.04,"15min":3.04},"last":0.443}]},{"osd":1,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":10.198,"5min":10.198,"15min":10.198},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":10.215999999999999,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64100000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.628}]},{"osd":7,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.69399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:19:49.110 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:19:49.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.261+0000 7fc00e19a700 1 Processor -- start 2023-11-24T06:19:49.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.262+0000 7fc00e19a700 1 -- start start 2023-11-24T06:19:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.262+0000 7fc00e19a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.262+0000 7fc00e19a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc00806c540 0x7fc0080b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.262+0000 7fc00e19a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0080af950 con 0x7fc0081446a0 2023-11-24T06:19:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.262+0000 7fc00e19a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc008144a80 con 0x7fc00806c540 2023-11-24T06:19:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.262+0000 7fc00e19a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00806c9b0 con 0x7fc0080adf00 2023-11-24T06:19:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc007fff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc00806c540 0x7fc0080b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc006ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc006ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00806c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:45830/0 (socket says 172.21.15.89:45830) 2023-11-24T06:19:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc006ffd700 1 -- 172.21.15.89:0/3229472800 learned_addr learned my addr 172.21.15.89:0/3229472800 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc0077fe700 1 --1- 172.21.15.89:0/3229472800 >> v1:172.21.15.123:6789/0 conn(0x7fc0081446a0 0x7fc0080b5ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:44278/0 (socket says 172.21.15.89:44278) 2023-11-24T06:19:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc006ffd700 1 -- 172.21.15.89:0/3229472800 >> v2:172.21.15.123:3300/0 conn(0x7fc00806c540 msgr2=0x7fc0080b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc006ffd700 1 --2- 172.21.15.89:0/3229472800 >> v2:172.21.15.123:3300/0 conn(0x7fc00806c540 0x7fc0080b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc006ffd700 1 -- 172.21.15.89:0/3229472800 >> v1:172.21.15.123:6789/0 conn(0x7fc0081446a0 legacy=0x7fc0080b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc006ffd700 1 -- 172.21.15.89:0/3229472800 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0080afc30 con 0x7fc0080adf00 2023-11-24T06:19:49.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc007fff700 1 --2- 172.21.15.89:0/3229472800 >> v2:172.21.15.123:3300/0 conn(0x7fc00806c540 0x7fc0080b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:49.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc006ffd700 1 --2- 172.21.15.89:0/3229472800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00806c000 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbff00098f0 tx=0x7fbff0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88e3f1b636965891 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:49.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc0067fc700 1 -- 172.21.15.89:0/3229472800 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0011830 con 0x7fc0080adf00 2023-11-24T06:19:49.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.263+0000 7fc0067fc700 1 -- 172.21.15.89:0/3229472800 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff0011990 con 0x7fc0080adf00 2023-11-24T06:19:49.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc0067fc700 1 -- 172.21.15.89:0/3229472800 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0011d20 con 0x7fc0080adf00 2023-11-24T06:19:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc00e19a700 1 -- 172.21.15.89:0/3229472800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 msgr2=0x7fc00806c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc00e19a700 1 --2- 172.21.15.89:0/3229472800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00806c000 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbff00098f0 tx=0x7fbff0009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc00e19a700 1 -- 172.21.15.89:0/3229472800 shutdown_connections 2023-11-24T06:19:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc00e19a700 1 --2- 172.21.15.89:0/3229472800 >> v2:172.21.15.123:3300/0 conn(0x7fc00806c540 0x7fc0080b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc00e19a700 1 --2- 172.21.15.89:0/3229472800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00806c000 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc00e19a700 1 -- 172.21.15.89:0/3229472800 >> 172.21.15.89:0/3229472800 conn(0x7fc008071b90 msgr2=0x7fc008071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc00e19a700 1 -- 172.21.15.89:0/3229472800 shutdown_connections 2023-11-24T06:19:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.264+0000 7fc00e19a700 1 -- 172.21.15.89:0/3229472800 wait complete. 2023-11-24T06:19:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.265+0000 7fc00e19a700 1 Processor -- start 2023-11-24T06:19:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.265+0000 7fc00e19a700 1 -- start start 2023-11-24T06:19:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc00e19a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00806c540 0x7fc0080d0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc00e19a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00809aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc00e19a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0081446a0 0x7fc00809af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc00e19a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080cdf90 con 0x7fc00806c540 2023-11-24T06:19:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc00e19a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080d06d0 con 0x7fc0081446a0 2023-11-24T06:19:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc00e19a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc008139c30 con 0x7fc0080adf00 2023-11-24T06:19:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc0077fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00806c540 0x7fc0080d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc0077fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00806c540 0x7fc0080d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36958/0 (socket says 172.21.15.89:36958) 2023-11-24T06:19:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc0077fe700 1 -- 172.21.15.89:0/2641439826 learned_addr learned my addr 172.21.15.89:0/2641439826 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc006ffd700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00809aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.266+0000 7fc007fff700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0081446a0 0x7fc00809af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc0077fe700 1 -- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 msgr2=0x7fc00809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc0077fe700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00809aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc0077fe700 1 -- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0081446a0 msgr2=0x7fc00809af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc0077fe700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0081446a0 0x7fc00809af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc0077fe700 1 -- 172.21.15.89:0/2641439826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00000bc80 con 0x7fc00806c540 2023-11-24T06:19:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc006ffd700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc007fff700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0081446a0 0x7fc00809af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc0077fe700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00806c540 0x7fc0080d0190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c520 tx=0x7fbff800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fbffffff700 1 -- 172.21.15.89:0/2641439826 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff8005cb0 con 0x7fc00806c540 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fbffffff700 1 -- 172.21.15.89:0/2641439826 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff8005e10 con 0x7fc00806c540 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff0009550 con 0x7fc00806c540 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fbffffff700 1 -- 172.21.15.89:0/2641439826 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff80072d0 con 0x7fc00806c540 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.267+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00813a1f0 con 0x7fc00806c540 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.268+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0080753f0 con 0x7fc00806c540 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.268+0000 7fbffffff700 1 -- 172.21.15.89:0/2641439826 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbff8007430 con 0x7fc00806c540 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.268+0000 7fbffffff700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbfe806aed0 0x7fbfe806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.269+0000 7fbffffff700 1 -- 172.21.15.89:0/2641439826 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbff8088950 con 0x7fc00806c540 2023-11-24T06:19:49.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.269+0000 7fc006ffd700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbfe806aed0 0x7fbfe806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.269+0000 7fc006ffd700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbfe806aed0 0x7fbfe806d390 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000c00 tx=0x7fbff0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:49.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.273+0000 7fbffffff700 1 -- 172.21.15.89:0/2641439826 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbff8053b40 con 0x7fc00806c540 2023-11-24T06:19:49.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.456+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0080b04d0 con 0x7fbfe806aed0 2023-11-24T06:19:49.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.464+0000 7fbffffff700 1 -- 172.21.15.89:0/2641439826 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289469 (secure 0 0 0) 0x7fbff0011c00 con 0x7fbfe806aed0 2023-11-24T06:19:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbfe806aed0 msgr2=0x7fbfe806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbfe806aed0 0x7fbfe806d390 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000c00 tx=0x7fbff0008030 comp rx=0 tx=0).stop 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00806c540 msgr2=0x7fc0080d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00806c540 0x7fc0080d0190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c520 tx=0x7fbff800c8e0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 shutdown_connections 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbfe806aed0 0x7fbfe806d390 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0081446a0 0x7fc00809af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00806c540 0x7fc0080d0190 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 --2- 172.21.15.89:0/2641439826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0080adf00 0x7fc00809aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 >> 172.21.15.89:0/2641439826 conn(0x7fc008071b90 msgr2=0x7fc00815d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 shutdown_connections 2023-11-24T06:19:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.467+0000 7fc00e19a700 1 -- 172.21.15.89:0/2641439826 wait complete. 2023-11-24T06:19:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:19:49.482 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2023-11-24T06:19:47.731509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220384,"kb_used_data":4196,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754336,"statfs":{"total":773094113280,"available":772868440064,"internally_reserved":0,"allocated":4296704,"data_stored":2375656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.146883"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624830+0000","last_change":"2023-11-24T06:19:44.620576+0000","last_active":"2023-11-24T06:19:44.624830+0000","last_peered":"2023-11-24T06:19:44.624830+0000","last_clean":"2023-11-24T06:19:44.624830+0000","last_became_active":"2023-11-24T06:19:44.620261+0000","last_became_peered":"2023-11-24T06:19:44.620261+0000","last_unstale":"2023-11-24T06:19:44.624830+0000","last_undegraded":"2023-11-24T06:19:44.624830+0000","last_fullsized":"2023-11-24T06:19:44.624830+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:07.005490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604933+0000","last_change":"2023-11-24T06:19:44.617073+0000","last_active":"2023-11-24T06:19:45.604933+0000","last_peered":"2023-11-24T06:19:45.604933+0000","last_clean":"2023-11-24T06:19:45.604933+0000","last_became_active":"2023-11-24T06:19:44.616814+0000","last_became_peered":"2023-11-24T06:19:44.616814+0000","last_unstale":"2023-11-24T06:19:45.604933+0000","last_undegraded":"2023-11-24T06:19:45.604933+0000","last_fullsized":"2023-11-24T06:19:45.604933+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:22:35.277438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603717+0000","last_change":"2023-11-24T06:19:44.620975+0000","last_active":"2023-11-24T06:19:45.603717+0000","last_peered":"2023-11-24T06:19:45.603717+0000","last_clean":"2023-11-24T06:19:45.603717+0000","last_became_active":"2023-11-24T06:19:44.620500+0000","last_became_peered":"2023-11-24T06:19:44.620500+0000","last_unstale":"2023-11-24T06:19:45.603717+0000","last_undegraded":"2023-11-24T06:19:45.603717+0000","last_fullsized":"2023-11-24T06:19:45.603717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:22.578783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106954+0000","last_change":"2023-11-24T06:19:44.620120+0000","last_active":"2023-11-24T06:19:46.106954+0000","last_peered":"2023-11-24T06:19:46.106954+0000","last_clean":"2023-11-24T06:19:46.106954+0000","last_became_active":"2023-11-24T06:19:44.619797+0000","last_became_peered":"2023-11-24T06:19:44.619797+0000","last_unstale":"2023-11-24T06:19:46.106954+0000","last_undegraded":"2023-11-24T06:19:46.106954+0000","last_fullsized":"2023-11-24T06:19:46.106954+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:09:27.389299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144468+0000","last_change":"2023-11-24T06:19:44.607813+0000","last_active":"2023-11-24T06:19:46.144468+0000","last_peered":"2023-11-24T06:19:46.144468+0000","last_clean":"2023-11-24T06:19:46.144468+0000","last_became_active":"2023-11-24T06:19:44.607521+0000","last_became_peered":"2023-11-24T06:19:44.607521+0000","last_unstale":"2023-11-24T06:19:46.144468+0000","last_undegraded":"2023-11-24T06:19:46.144468+0000","last_fullsized":"2023-11-24T06:19:46.144468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:03.428940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604755+0000","last_change":"2023-11-24T06:19:44.617997+0000","last_active":"2023-11-24T06:19:45.604755+0000","last_peered":"2023-11-24T06:19:45.604755+0000","last_clean":"2023-11-24T06:19:45.604755+0000","last_became_active":"2023-11-24T06:19:44.617699+0000","last_became_peered":"2023-11-24T06:19:44.617699+0000","last_unstale":"2023-11-24T06:19:45.604755+0000","last_undegraded":"2023-11-24T06:19:45.604755+0000","last_fullsized":"2023-11-24T06:19:45.604755+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:17.149210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604218+0000","last_change":"2023-11-24T06:19:44.615653+0000","last_active":"2023-11-24T06:19:45.604218+0000","last_peered":"2023-11-24T06:19:45.604218+0000","last_clean":"2023-11-24T06:19:45.604218+0000","last_became_active":"2023-11-24T06:19:44.615411+0000","last_became_peered":"2023-11-24T06:19:44.615411+0000","last_unstale":"2023-11-24T06:19:45.604218+0000","last_undegraded":"2023-11-24T06:19:45.604218+0000","last_fullsized":"2023-11-24T06:19:45.604218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:54.283680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603821+0000","last_change":"2023-11-24T06:19:44.605475+0000","last_active":"2023-11-24T06:19:45.603821+0000","last_peered":"2023-11-24T06:19:45.603821+0000","last_clean":"2023-11-24T06:19:45.603821+0000","last_became_active":"2023-11-24T06:19:44.605100+0000","last_became_peered":"2023-11-24T06:19:44.605100+0000","last_unstale":"2023-11-24T06:19:45.603821+0000","last_undegraded":"2023-11-24T06:19:45.603821+0000","last_fullsized":"2023-11-24T06:19:45.603821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:22.578783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.621965+0000","last_change":"2023-11-24T06:19:44.615217+0000","last_active":"2023-11-24T06:19:44.621965+0000","last_peered":"2023-11-24T06:19:44.621965+0000","last_clean":"2023-11-24T06:19:44.621965+0000","last_became_active":"2023-11-24T06:19:44.613739+0000","last_became_peered":"2023-11-24T06:19:44.613739+0000","last_unstale":"2023-11-24T06:19:44.621965+0000","last_undegraded":"2023-11-24T06:19:44.621965+0000","last_fullsized":"2023-11-24T06:19:44.621965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:59:36.466491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107232+0000","last_change":"2023-11-24T06:19:44.614457+0000","last_active":"2023-11-24T06:19:46.107232+0000","last_peered":"2023-11-24T06:19:46.107232+0000","last_clean":"2023-11-24T06:19:46.107232+0000","last_became_active":"2023-11-24T06:19:44.614123+0000","last_became_peered":"2023-11-24T06:19:44.614123+0000","last_unstale":"2023-11-24T06:19:46.107232+0000","last_undegraded":"2023-11-24T06:19:46.107232+0000","last_fullsized":"2023-11-24T06:19:46.107232+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:11:48.592746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603724+0000","last_change":"2023-11-24T06:19:44.611487+0000","last_active":"2023-11-24T06:19:45.603724+0000","last_peered":"2023-11-24T06:19:45.603724+0000","last_clean":"2023-11-24T06:19:45.603724+0000","last_became_active":"2023-11-24T06:19:44.610852+0000","last_became_peered":"2023-11-24T06:19:44.610852+0000","last_unstale":"2023-11-24T06:19:45.603724+0000","last_undegraded":"2023-11-24T06:19:45.603724+0000","last_fullsized":"2023-11-24T06:19:45.603724+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:04:44.567943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603612+0000","last_change":"2023-11-24T06:19:44.621311+0000","last_active":"2023-11-24T06:19:45.603612+0000","last_peered":"2023-11-24T06:19:45.603612+0000","last_clean":"2023-11-24T06:19:45.603612+0000","last_became_active":"2023-11-24T06:19:44.619835+0000","last_became_peered":"2023-11-24T06:19:44.619835+0000","last_unstale":"2023-11-24T06:19:45.603612+0000","last_undegraded":"2023-11-24T06:19:45.603612+0000","last_fullsized":"2023-11-24T06:19:45.603612+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:04:44.567943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144597+0000","last_change":"2023-11-24T06:19:44.604646+0000","last_active":"2023-11-24T06:19:46.144597+0000","last_peered":"2023-11-24T06:19:46.144597+0000","last_clean":"2023-11-24T06:19:46.144597+0000","last_became_active":"2023-11-24T06:19:44.604275+0000","last_became_peered":"2023-11-24T06:19:44.604275+0000","last_unstale":"2023-11-24T06:19:46.144597+0000","last_undegraded":"2023-11-24T06:19:46.144597+0000","last_fullsized":"2023-11-24T06:19:46.144597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:57.374633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604704+0000","last_change":"2023-11-24T06:19:44.605526+0000","last_active":"2023-11-24T06:19:45.604704+0000","last_peered":"2023-11-24T06:19:45.604704+0000","last_clean":"2023-11-24T06:19:45.604704+0000","last_became_active":"2023-11-24T06:19:44.605255+0000","last_became_peered":"2023-11-24T06:19:44.605255+0000","last_unstale":"2023-11-24T06:19:45.604704+0000","last_undegraded":"2023-11-24T06:19:45.604704+0000","last_fullsized":"2023-11-24T06:19:45.604704+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:34.436870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603211+0000","last_change":"2023-11-24T06:19:44.621768+0000","last_active":"2023-11-24T06:19:45.603211+0000","last_peered":"2023-11-24T06:19:45.603211+0000","last_clean":"2023-11-24T06:19:45.603211+0000","last_became_active":"2023-11-24T06:19:44.621481+0000","last_became_peered":"2023-11-24T06:19:44.621481+0000","last_unstale":"2023-11-24T06:19:45.603211+0000","last_undegraded":"2023-11-24T06:19:45.603211+0000","last_fullsized":"2023-11-24T06:19:45.603211+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:17.543360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604002+0000","last_change":"2023-11-24T06:19:44.611740+0000","last_active":"2023-11-24T06:19:45.604002+0000","last_peered":"2023-11-24T06:19:45.604002+0000","last_clean":"2023-11-24T06:19:45.604002+0000","last_became_active":"2023-11-24T06:19:44.611350+0000","last_became_peered":"2023-11-24T06:19:44.611350+0000","last_unstale":"2023-11-24T06:19:45.604002+0000","last_undegraded":"2023-11-24T06:19:45.604002+0000","last_fullsized":"2023-11-24T06:19:45.604002+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:35:04.205238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603943+0000","last_change":"2023-11-24T06:19:44.619948+0000","last_active":"2023-11-24T06:19:45.603943+0000","last_peered":"2023-11-24T06:19:45.603943+0000","last_clean":"2023-11-24T06:19:45.603943+0000","last_became_active":"2023-11-24T06:19:44.619146+0000","last_became_peered":"2023-11-24T06:19:44.619146+0000","last_unstale":"2023-11-24T06:19:45.603943+0000","last_undegraded":"2023-11-24T06:19:45.603943+0000","last_fullsized":"2023-11-24T06:19:45.603943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:24.264848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605113+0000","last_change":"2023-11-24T06:19:44.610677+0000","last_active":"2023-11-24T06:19:45.605113+0000","last_peered":"2023-11-24T06:19:45.605113+0000","last_clean":"2023-11-24T06:19:45.605113+0000","last_became_active":"2023-11-24T06:19:44.609508+0000","last_became_peered":"2023-11-24T06:19:44.609508+0000","last_unstale":"2023-11-24T06:19:45.605113+0000","last_undegraded":"2023-11-24T06:19:45.605113+0000","last_fullsized":"2023-11-24T06:19:45.605113+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:26:06.485092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604804+0000","last_change":"2023-11-24T06:19:44.608764+0000","last_active":"2023-11-24T06:19:45.604804+0000","last_peered":"2023-11-24T06:19:45.604804+0000","last_clean":"2023-11-24T06:19:45.604804+0000","last_became_active":"2023-11-24T06:19:44.608472+0000","last_became_peered":"2023-11-24T06:19:44.608472+0000","last_unstale":"2023-11-24T06:19:45.604804+0000","last_undegraded":"2023-11-24T06:19:45.604804+0000","last_fullsized":"2023-11-24T06:19:45.604804+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:41:17.872131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144529+0000","last_change":"2023-11-24T06:19:44.617798+0000","last_active":"2023-11-24T06:19:46.144529+0000","last_peered":"2023-11-24T06:19:46.144529+0000","last_clean":"2023-11-24T06:19:46.144529+0000","last_became_active":"2023-11-24T06:19:44.617465+0000","last_became_peered":"2023-11-24T06:19:44.617465+0000","last_unstale":"2023-11-24T06:19:46.144529+0000","last_undegraded":"2023-11-24T06:19:46.144529+0000","last_fullsized":"2023-11-24T06:19:46.144529+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:11.876249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624530+0000","last_change":"2023-11-24T06:19:44.617922+0000","last_active":"2023-11-24T06:19:44.624530+0000","last_peered":"2023-11-24T06:19:44.624530+0000","last_clean":"2023-11-24T06:19:44.624530+0000","last_became_active":"2023-11-24T06:19:44.617621+0000","last_became_peered":"2023-11-24T06:19:44.617621+0000","last_unstale":"2023-11-24T06:19:44.624530+0000","last_undegraded":"2023-11-24T06:19:44.624530+0000","last_fullsized":"2023-11-24T06:19:44.624530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:27.706327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.622077+0000","last_change":"2023-11-24T06:19:44.615852+0000","last_active":"2023-11-24T06:19:44.622077+0000","last_peered":"2023-11-24T06:19:44.622077+0000","last_clean":"2023-11-24T06:19:44.622077+0000","last_became_active":"2023-11-24T06:19:44.615557+0000","last_became_peered":"2023-11-24T06:19:44.615557+0000","last_unstale":"2023-11-24T06:19:44.622077+0000","last_undegraded":"2023-11-24T06:19:44.622077+0000","last_fullsized":"2023-11-24T06:19:44.622077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:50:25.208584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624267+0000","last_change":"2023-11-24T06:19:44.612165+0000","last_active":"2023-11-24T06:19:46.624267+0000","last_peered":"2023-11-24T06:19:46.624267+0000","last_clean":"2023-11-24T06:19:46.624267+0000","last_became_active":"2023-11-24T06:19:44.611888+0000","last_became_peered":"2023-11-24T06:19:44.611888+0000","last_unstale":"2023-11-24T06:19:46.624267+0000","last_undegraded":"2023-11-24T06:19:46.624267+0000","last_fullsized":"2023-11-24T06:19:46.624267+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:21.515369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603370+0000","last_change":"2023-11-24T06:19:44.617293+0000","last_active":"2023-11-24T06:19:45.603370+0000","last_peered":"2023-11-24T06:19:45.603370+0000","last_clean":"2023-11-24T06:19:45.603370+0000","last_became_active":"2023-11-24T06:19:44.616755+0000","last_became_peered":"2023-11-24T06:19:44.616755+0000","last_unstale":"2023-11-24T06:19:45.603370+0000","last_undegraded":"2023-11-24T06:19:45.603370+0000","last_fullsized":"2023-11-24T06:19:45.603370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:11:48.592746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107367+0000","last_change":"2023-11-24T06:19:44.608188+0000","last_active":"2023-11-24T06:19:46.107367+0000","last_peered":"2023-11-24T06:19:46.107367+0000","last_clean":"2023-11-24T06:19:46.107367+0000","last_became_active":"2023-11-24T06:19:44.607822+0000","last_became_peered":"2023-11-24T06:19:44.607822+0000","last_unstale":"2023-11-24T06:19:46.107367+0000","last_undegraded":"2023-11-24T06:19:46.107367+0000","last_fullsized":"2023-11-24T06:19:46.107367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:04:44.567943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605147+0000","last_change":"2023-11-24T06:19:44.609651+0000","last_active":"2023-11-24T06:19:45.605147+0000","last_peered":"2023-11-24T06:19:45.605147+0000","last_clean":"2023-11-24T06:19:45.605147+0000","last_became_active":"2023-11-24T06:19:44.607936+0000","last_became_peered":"2023-11-24T06:19:44.607936+0000","last_unstale":"2023-11-24T06:19:45.605147+0000","last_undegraded":"2023-11-24T06:19:45.605147+0000","last_fullsized":"2023-11-24T06:19:45.605147+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:10:59.958794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107201+0000","last_change":"2023-11-24T06:19:44.616238+0000","last_active":"2023-11-24T06:19:46.107201+0000","last_peered":"2023-11-24T06:19:46.107201+0000","last_clean":"2023-11-24T06:19:46.107201+0000","last_became_active":"2023-11-24T06:19:44.615960+0000","last_became_peered":"2023-11-24T06:19:44.615960+0000","last_unstale":"2023-11-24T06:19:46.107201+0000","last_undegraded":"2023-11-24T06:19:46.107201+0000","last_fullsized":"2023-11-24T06:19:46.107201+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:43.733250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603725+0000","last_change":"2023-11-24T06:19:44.617127+0000","last_active":"2023-11-24T06:19:45.603725+0000","last_peered":"2023-11-24T06:19:45.603725+0000","last_clean":"2023-11-24T06:19:45.603725+0000","last_became_active":"2023-11-24T06:19:44.616734+0000","last_became_peered":"2023-11-24T06:19:44.616734+0000","last_unstale":"2023-11-24T06:19:45.603725+0000","last_undegraded":"2023-11-24T06:19:45.603725+0000","last_fullsized":"2023-11-24T06:19:45.603725+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:58:50.312633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.622466+0000","last_change":"2023-11-24T06:19:44.616104+0000","last_active":"2023-11-24T06:19:44.622466+0000","last_peered":"2023-11-24T06:19:44.622466+0000","last_clean":"2023-11-24T06:19:44.622466+0000","last_became_active":"2023-11-24T06:19:44.615221+0000","last_became_peered":"2023-11-24T06:19:44.615221+0000","last_unstale":"2023-11-24T06:19:44.622466+0000","last_undegraded":"2023-11-24T06:19:44.622466+0000","last_fullsized":"2023-11-24T06:19:44.622466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:00:23.279015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604143+0000","last_change":"2023-11-24T06:19:44.618110+0000","last_active":"2023-11-24T06:19:45.604143+0000","last_peered":"2023-11-24T06:19:45.604143+0000","last_clean":"2023-11-24T06:19:45.604143+0000","last_became_active":"2023-11-24T06:19:44.617695+0000","last_became_peered":"2023-11-24T06:19:44.617695+0000","last_unstale":"2023-11-24T06:19:45.604143+0000","last_undegraded":"2023-11-24T06:19:45.604143+0000","last_fullsized":"2023-11-24T06:19:45.604143+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:41:17.872131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624885+0000","last_change":"2023-11-24T06:19:44.621432+0000","last_active":"2023-11-24T06:19:44.624885+0000","last_peered":"2023-11-24T06:19:44.624885+0000","last_clean":"2023-11-24T06:19:44.624885+0000","last_became_active":"2023-11-24T06:19:44.618113+0000","last_became_peered":"2023-11-24T06:19:44.618113+0000","last_unstale":"2023-11-24T06:19:44.624885+0000","last_undegraded":"2023-11-24T06:19:44.624885+0000","last_fullsized":"2023-11-24T06:19:44.624885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:57.374633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604110+0000","last_change":"2023-11-24T06:19:44.619350+0000","last_active":"2023-11-24T06:19:45.604110+0000","last_peered":"2023-11-24T06:19:45.604110+0000","last_clean":"2023-11-24T06:19:45.604110+0000","last_became_active":"2023-11-24T06:19:44.618892+0000","last_became_peered":"2023-11-24T06:19:44.618892+0000","last_unstale":"2023-11-24T06:19:45.604110+0000","last_undegraded":"2023-11-24T06:19:45.604110+0000","last_fullsized":"2023-11-24T06:19:45.604110+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:17.149210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624461+0000","last_change":"2023-11-24T06:19:44.612782+0000","last_active":"2023-11-24T06:19:46.624461+0000","last_peered":"2023-11-24T06:19:46.624461+0000","last_clean":"2023-11-24T06:19:46.624461+0000","last_became_active":"2023-11-24T06:19:44.612491+0000","last_became_peered":"2023-11-24T06:19:44.612491+0000","last_unstale":"2023-11-24T06:19:46.624461+0000","last_undegraded":"2023-11-24T06:19:46.624461+0000","last_fullsized":"2023-11-24T06:19:46.624461+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:07:41.777438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.993094+0000","last_change":"2023-11-24T06:19:43.603470+0000","last_active":"2023-11-24T06:19:44.993094+0000","last_peered":"2023-11-24T06:19:44.993094+0000","last_clean":"2023-11-24T06:19:44.993094+0000","last_became_active":"2023-11-24T06:19:43.603222+0000","last_became_peered":"2023-11-24T06:19:43.603222+0000","last_unstale":"2023-11-24T06:19:44.993094+0000","last_undegraded":"2023-11-24T06:19:44.993094+0000","last_fullsized":"2023-11-24T06:19:44.993094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:05.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.627205+0000","last_change":"2023-11-24T06:19:44.622557+0000","last_active":"2023-11-24T06:19:44.627205+0000","last_peered":"2023-11-24T06:19:44.627205+0000","last_clean":"2023-11-24T06:19:44.627205+0000","last_became_active":"2023-11-24T06:19:44.622258+0000","last_became_peered":"2023-11-24T06:19:44.622258+0000","last_unstale":"2023-11-24T06:19:44.627205+0000","last_undegraded":"2023-11-24T06:19:44.627205+0000","last_fullsized":"2023-11-24T06:19:44.627205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:11.876249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624434+0000","last_change":"2023-11-24T06:19:43.601904+0000","last_active":"2023-11-24T06:19:46.624434+0000","last_peered":"2023-11-24T06:19:46.624434+0000","last_clean":"2023-11-24T06:19:46.624434+0000","last_became_active":"2023-11-24T06:19:43.601620+0000","last_became_peered":"2023-11-24T06:19:43.601620+0000","last_unstale":"2023-11-24T06:19:46.624434+0000","last_undegraded":"2023-11-24T06:19:46.624434+0000","last_fullsized":"2023-11-24T06:19:46.624434+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:05.997314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624414+0000","last_change":"2023-11-24T06:19:44.616895+0000","last_active":"2023-11-24T06:19:44.624414+0000","last_peered":"2023-11-24T06:19:44.624414+0000","last_clean":"2023-11-24T06:19:44.624414+0000","last_became_active":"2023-11-24T06:19:44.616610+0000","last_became_peered":"2023-11-24T06:19:44.616610+0000","last_unstale":"2023-11-24T06:19:44.624414+0000","last_undegraded":"2023-11-24T06:19:44.624414+0000","last_fullsized":"2023-11-24T06:19:44.624414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:01.710358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604745+0000","last_change":"2023-11-24T06:19:43.607012+0000","last_active":"2023-11-24T06:19:45.604745+0000","last_peered":"2023-11-24T06:19:45.604745+0000","last_clean":"2023-11-24T06:19:45.604745+0000","last_became_active":"2023-11-24T06:19:43.606729+0000","last_became_peered":"2023-11-24T06:19:43.606729+0000","last_unstale":"2023-11-24T06:19:45.604745+0000","last_undegraded":"2023-11-24T06:19:45.604745+0000","last_fullsized":"2023-11-24T06:19:45.604745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:11:47.584570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107212+0000","last_change":"2023-11-24T06:19:44.616126+0000","last_active":"2023-11-24T06:19:46.107212+0000","last_peered":"2023-11-24T06:19:46.107212+0000","last_clean":"2023-11-24T06:19:46.107212+0000","last_became_active":"2023-11-24T06:19:44.615730+0000","last_became_peered":"2023-11-24T06:19:44.615730+0000","last_unstale":"2023-11-24T06:19:46.107212+0000","last_undegraded":"2023-11-24T06:19:46.107212+0000","last_fullsized":"2023-11-24T06:19:46.107212+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:17.543360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144559+0000","last_change":"2023-11-24T06:19:43.608044+0000","last_active":"2023-11-24T06:19:46.144559+0000","last_peered":"2023-11-24T06:19:46.144559+0000","last_clean":"2023-11-24T06:19:46.144559+0000","last_became_active":"2023-11-24T06:19:43.607808+0000","last_became_peered":"2023-11-24T06:19:43.607808+0000","last_unstale":"2023-11-24T06:19:46.144559+0000","last_undegraded":"2023-11-24T06:19:46.144559+0000","last_fullsized":"2023-11-24T06:19:46.144559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:28:26.698151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603696+0000","last_change":"2023-11-24T06:19:44.617935+0000","last_active":"2023-11-24T06:19:45.603696+0000","last_peered":"2023-11-24T06:19:45.603696+0000","last_clean":"2023-11-24T06:19:45.603696+0000","last_became_active":"2023-11-24T06:19:44.617352+0000","last_became_peered":"2023-11-24T06:19:44.617352+0000","last_unstale":"2023-11-24T06:19:45.603696+0000","last_undegraded":"2023-11-24T06:19:45.603696+0000","last_fullsized":"2023-11-24T06:19:45.603696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:11:48.592746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.992772+0000","last_change":"2023-11-24T06:19:43.599161+0000","last_active":"2023-11-24T06:19:44.992772+0000","last_peered":"2023-11-24T06:19:44.992772+0000","last_clean":"2023-11-24T06:19:44.992772+0000","last_became_active":"2023-11-24T06:19:43.598849+0000","last_became_peered":"2023-11-24T06:19:43.598849+0000","last_unstale":"2023-11-24T06:19:44.992772+0000","last_undegraded":"2023-11-24T06:19:44.992772+0000","last_fullsized":"2023-11-24T06:19:44.992772+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:05.997314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603199+0000","last_change":"2023-11-24T06:19:44.619679+0000","last_active":"2023-11-24T06:19:45.603199+0000","last_peered":"2023-11-24T06:19:45.603199+0000","last_clean":"2023-11-24T06:19:45.603199+0000","last_became_active":"2023-11-24T06:19:44.619280+0000","last_became_peered":"2023-11-24T06:19:44.619280+0000","last_unstale":"2023-11-24T06:19:45.603199+0000","last_undegraded":"2023-11-24T06:19:45.603199+0000","last_fullsized":"2023-11-24T06:19:45.603199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:43.733250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.992818+0000","last_change":"2023-11-24T06:19:43.599222+0000","last_active":"2023-11-24T06:19:44.992818+0000","last_peered":"2023-11-24T06:19:44.992818+0000","last_clean":"2023-11-24T06:19:44.992818+0000","last_became_active":"2023-11-24T06:19:43.598908+0000","last_became_peered":"2023-11-24T06:19:43.598908+0000","last_unstale":"2023-11-24T06:19:44.992818+0000","last_undegraded":"2023-11-24T06:19:44.992818+0000","last_fullsized":"2023-11-24T06:19:44.992818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:51:12.352766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624687+0000","last_change":"2023-11-24T06:19:44.618829+0000","last_active":"2023-11-24T06:19:44.624687+0000","last_peered":"2023-11-24T06:19:44.624687+0000","last_clean":"2023-11-24T06:19:44.624687+0000","last_became_active":"2023-11-24T06:19:44.618507+0000","last_became_peered":"2023-11-24T06:19:44.618507+0000","last_unstale":"2023-11-24T06:19:44.624687+0000","last_undegraded":"2023-11-24T06:19:44.624687+0000","last_fullsized":"2023-11-24T06:19:44.624687+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:03.428940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604963+0000","last_change":"2023-11-24T06:19:43.605710+0000","last_active":"2023-11-24T06:19:45.604963+0000","last_peered":"2023-11-24T06:19:45.604963+0000","last_clean":"2023-11-24T06:19:45.604963+0000","last_became_active":"2023-11-24T06:19:43.605085+0000","last_became_peered":"2023-11-24T06:19:43.605085+0000","last_unstale":"2023-11-24T06:19:45.604963+0000","last_undegraded":"2023-11-24T06:19:45.604963+0000","last_fullsized":"2023-11-24T06:19:45.604963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.994336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106814+0000","last_change":"2023-11-24T06:19:44.618009+0000","last_active":"2023-11-24T06:19:46.106814+0000","last_peered":"2023-11-24T06:19:46.106814+0000","last_clean":"2023-11-24T06:19:46.106814+0000","last_became_active":"2023-11-24T06:19:44.617747+0000","last_became_peered":"2023-11-24T06:19:44.617747+0000","last_unstale":"2023-11-24T06:19:46.106814+0000","last_undegraded":"2023-11-24T06:19:46.106814+0000","last_fullsized":"2023-11-24T06:19:46.106814+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:51:22.442670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.609487+0000","last_change":"2023-11-24T06:19:43.594541+0000","last_active":"2023-11-24T06:19:44.609487+0000","last_peered":"2023-11-24T06:19:44.609487+0000","last_clean":"2023-11-24T06:19:44.609487+0000","last_became_active":"2023-11-24T06:19:43.594258+0000","last_became_peered":"2023-11-24T06:19:43.594258+0000","last_unstale":"2023-11-24T06:19:44.609487+0000","last_undegraded":"2023-11-24T06:19:44.609487+0000","last_fullsized":"2023-11-24T06:19:44.609487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:10.868073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603922+0000","last_change":"2023-11-24T06:19:44.620214+0000","last_active":"2023-11-24T06:19:45.603922+0000","last_peered":"2023-11-24T06:19:45.603922+0000","last_clean":"2023-11-24T06:19:45.603922+0000","last_became_active":"2023-11-24T06:19:44.619768+0000","last_became_peered":"2023-11-24T06:19:44.619768+0000","last_unstale":"2023-11-24T06:19:45.603922+0000","last_undegraded":"2023-11-24T06:19:45.603922+0000","last_fullsized":"2023-11-24T06:19:45.603922+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:35:04.205238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604887+0000","last_change":"2023-11-24T06:19:43.603672+0000","last_active":"2023-11-24T06:19:45.604887+0000","last_peered":"2023-11-24T06:19:45.604887+0000","last_clean":"2023-11-24T06:19:45.604887+0000","last_became_active":"2023-11-24T06:19:43.602527+0000","last_became_peered":"2023-11-24T06:19:43.602527+0000","last_unstale":"2023-11-24T06:19:45.604887+0000","last_undegraded":"2023-11-24T06:19:45.604887+0000","last_fullsized":"2023-11-24T06:19:45.604887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:04:43.559768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605063+0000","last_change":"2023-11-24T06:19:44.609926+0000","last_active":"2023-11-24T06:19:45.605063+0000","last_peered":"2023-11-24T06:19:45.605063+0000","last_clean":"2023-11-24T06:19:45.605063+0000","last_became_active":"2023-11-24T06:19:44.609653+0000","last_became_peered":"2023-11-24T06:19:44.609653+0000","last_unstale":"2023-11-24T06:19:45.605063+0000","last_undegraded":"2023-11-24T06:19:45.605063+0000","last_fullsized":"2023-11-24T06:19:45.605063+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:53:46.166270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624511+0000","last_change":"2023-11-24T06:19:43.606931+0000","last_active":"2023-11-24T06:19:46.624511+0000","last_peered":"2023-11-24T06:19:46.624511+0000","last_clean":"2023-11-24T06:19:46.624511+0000","last_became_active":"2023-11-24T06:19:43.606619+0000","last_became_peered":"2023-11-24T06:19:43.606619+0000","last_unstale":"2023-11-24T06:19:46.624511+0000","last_undegraded":"2023-11-24T06:19:46.624511+0000","last_fullsized":"2023-11-24T06:19:46.624511+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:56.366458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604855+0000","last_change":"2023-11-24T06:19:44.605015+0000","last_active":"2023-11-24T06:19:45.604855+0000","last_peered":"2023-11-24T06:19:45.604855+0000","last_clean":"2023-11-24T06:19:45.604855+0000","last_became_active":"2023-11-24T06:19:44.604692+0000","last_became_peered":"2023-11-24T06:19:44.604692+0000","last_unstale":"2023-11-24T06:19:45.604855+0000","last_undegraded":"2023-11-24T06:19:45.604855+0000","last_fullsized":"2023-11-24T06:19:45.604855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:54.283680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603980+0000","last_change":"2023-11-24T06:19:43.602959+0000","last_active":"2023-11-24T06:19:45.603980+0000","last_peered":"2023-11-24T06:19:45.603980+0000","last_clean":"2023-11-24T06:19:45.603980+0000","last_became_active":"2023-11-24T06:19:43.602599+0000","last_became_peered":"2023-11-24T06:19:43.602599+0000","last_unstale":"2023-11-24T06:19:45.603980+0000","last_undegraded":"2023-11-24T06:19:45.603980+0000","last_fullsized":"2023-11-24T06:19:45.603980+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:55:10.412916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603873+0000","last_change":"2023-11-24T06:19:44.620784+0000","last_active":"2023-11-24T06:19:45.603873+0000","last_peered":"2023-11-24T06:19:45.603873+0000","last_clean":"2023-11-24T06:19:45.603873+0000","last_became_active":"2023-11-24T06:19:44.620154+0000","last_became_peered":"2023-11-24T06:19:44.620154+0000","last_unstale":"2023-11-24T06:19:45.603873+0000","last_undegraded":"2023-11-24T06:19:45.603873+0000","last_fullsized":"2023-11-24T06:19:45.603873+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:45:24.264848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605006+0000","last_change":"2023-11-24T06:19:43.607375+0000","last_active":"2023-11-24T06:19:45.605006+0000","last_peered":"2023-11-24T06:19:45.605006+0000","last_clean":"2023-11-24T06:19:45.605006+0000","last_became_active":"2023-11-24T06:19:43.607119+0000","last_became_peered":"2023-11-24T06:19:43.607119+0000","last_unstale":"2023-11-24T06:19:45.605006+0000","last_undegraded":"2023-11-24T06:19:45.605006+0000","last_fullsized":"2023-11-24T06:19:45.605006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:16.535184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106880+0000","last_change":"2023-11-24T06:19:44.618619+0000","last_active":"2023-11-24T06:19:46.106880+0000","last_peered":"2023-11-24T06:19:46.106880+0000","last_clean":"2023-11-24T06:19:46.106880+0000","last_became_active":"2023-11-24T06:19:44.618344+0000","last_became_peered":"2023-11-24T06:19:44.618344+0000","last_unstale":"2023-11-24T06:19:46.106880+0000","last_undegraded":"2023-11-24T06:19:46.106880+0000","last_fullsized":"2023-11-24T06:19:46.106880+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:55:11.421092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.991215+0000","last_change":"2023-11-24T06:19:43.597304+0000","last_active":"2023-11-24T06:19:44.991215+0000","last_peered":"2023-11-24T06:19:44.991215+0000","last_clean":"2023-11-24T06:19:44.991215+0000","last_became_active":"2023-11-24T06:19:43.597030+0000","last_became_peered":"2023-11-24T06:19:43.597030+0000","last_unstale":"2023-11-24T06:19:44.991215+0000","last_undegraded":"2023-11-24T06:19:44.991215+0000","last_fullsized":"2023-11-24T06:19:44.991215+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:36:45.636780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605142+0000","last_change":"2023-11-24T06:19:44.615720+0000","last_active":"2023-11-24T06:19:45.605142+0000","last_peered":"2023-11-24T06:19:45.605142+0000","last_clean":"2023-11-24T06:19:45.605142+0000","last_became_active":"2023-11-24T06:19:44.615442+0000","last_became_peered":"2023-11-24T06:19:44.615442+0000","last_unstale":"2023-11-24T06:19:45.605142+0000","last_undegraded":"2023-11-24T06:19:45.605142+0000","last_fullsized":"2023-11-24T06:19:45.605142+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:58.700838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106842+0000","last_change":"2023-11-24T06:19:43.602015+0000","last_active":"2023-11-24T06:19:46.106842+0000","last_peered":"2023-11-24T06:19:46.106842+0000","last_clean":"2023-11-24T06:19:46.106842+0000","last_became_active":"2023-11-24T06:19:43.601730+0000","last_became_peered":"2023-11-24T06:19:43.601730+0000","last_unstale":"2023-11-24T06:19:46.106842+0000","last_undegraded":"2023-11-24T06:19:46.106842+0000","last_fullsized":"2023-11-24T06:19:46.106842+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:10.868073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604131+0000","last_change":"2023-11-24T06:19:44.619158+0000","last_active":"2023-11-24T06:19:45.604131+0000","last_peered":"2023-11-24T06:19:45.604131+0000","last_clean":"2023-11-24T06:19:45.604131+0000","last_became_active":"2023-11-24T06:19:44.618560+0000","last_became_peered":"2023-11-24T06:19:44.618560+0000","last_unstale":"2023-11-24T06:19:45.604131+0000","last_undegraded":"2023-11-24T06:19:45.604131+0000","last_fullsized":"2023-11-24T06:19:45.604131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:34.436870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604710+0000","last_change":"2023-11-24T06:19:43.603182+0000","last_active":"2023-11-24T06:19:45.604710+0000","last_peered":"2023-11-24T06:19:45.604710+0000","last_clean":"2023-11-24T06:19:45.604710+0000","last_became_active":"2023-11-24T06:19:43.602910+0000","last_became_peered":"2023-11-24T06:19:43.602910+0000","last_unstale":"2023-11-24T06:19:45.604710+0000","last_undegraded":"2023-11-24T06:19:45.604710+0000","last_fullsized":"2023-11-24T06:19:45.604710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:45:42.725075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.627267+0000","last_change":"2023-11-24T06:19:44.623347+0000","last_active":"2023-11-24T06:19:44.627267+0000","last_peered":"2023-11-24T06:19:44.627267+0000","last_clean":"2023-11-24T06:19:44.627267+0000","last_became_active":"2023-11-24T06:19:44.623005+0000","last_became_peered":"2023-11-24T06:19:44.623005+0000","last_unstale":"2023-11-24T06:19:44.627267+0000","last_undegraded":"2023-11-24T06:19:44.627267+0000","last_fullsized":"2023-11-24T06:19:44.627267+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:25:57.248982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604431+0000","last_change":"2023-11-24T06:19:43.606229+0000","last_active":"2023-11-24T06:19:45.604431+0000","last_peered":"2023-11-24T06:19:45.604431+0000","last_clean":"2023-11-24T06:19:45.604431+0000","last_became_active":"2023-11-24T06:19:43.605970+0000","last_became_peered":"2023-11-24T06:19:43.605970+0000","last_unstale":"2023-11-24T06:19:45.604431+0000","last_undegraded":"2023-11-24T06:19:45.604431+0000","last_fullsized":"2023-11-24T06:19:45.604431+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:04:21.570607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.624987+0000","last_change":"2023-11-24T06:19:44.623354+0000","last_active":"2023-11-24T06:19:44.624987+0000","last_peered":"2023-11-24T06:19:44.624987+0000","last_clean":"2023-11-24T06:19:44.624987+0000","last_became_active":"2023-11-24T06:19:44.623070+0000","last_became_peered":"2023-11-24T06:19:44.623070+0000","last_unstale":"2023-11-24T06:19:44.624987+0000","last_undegraded":"2023-11-24T06:19:44.624987+0000","last_fullsized":"2023-11-24T06:19:44.624987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:25:57.248982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107619+0000","last_change":"2023-11-24T06:19:43.609838+0000","last_active":"2023-11-24T06:19:46.107619+0000","last_peered":"2023-11-24T06:19:46.107619+0000","last_clean":"2023-11-24T06:19:46.107619+0000","last_became_active":"2023-11-24T06:19:43.609553+0000","last_became_peered":"2023-11-24T06:19:43.609553+0000","last_unstale":"2023-11-24T06:19:46.107619+0000","last_undegraded":"2023-11-24T06:19:46.107619+0000","last_fullsized":"2023-11-24T06:19:46.107619+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:07:40.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604218+0000","last_change":"2023-11-24T06:19:44.623125+0000","last_active":"2023-11-24T06:19:45.604218+0000","last_peered":"2023-11-24T06:19:45.604218+0000","last_clean":"2023-11-24T06:19:45.604218+0000","last_became_active":"2023-11-24T06:19:44.622866+0000","last_became_peered":"2023-11-24T06:19:44.622866+0000","last_unstale":"2023-11-24T06:19:45.604218+0000","last_undegraded":"2023-11-24T06:19:45.604218+0000","last_fullsized":"2023-11-24T06:19:45.604218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:25:17.149210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.991123+0000","last_change":"2023-11-24T06:19:43.601747+0000","last_active":"2023-11-24T06:19:44.991123+0000","last_peered":"2023-11-24T06:19:44.991123+0000","last_clean":"2023-11-24T06:19:44.991123+0000","last_became_active":"2023-11-24T06:19:43.601458+0000","last_became_peered":"2023-11-24T06:19:43.601458+0000","last_unstale":"2023-11-24T06:19:44.991123+0000","last_undegraded":"2023-11-24T06:19:44.991123+0000","last_fullsized":"2023-11-24T06:19:44.991123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:56.366458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107052+0000","last_change":"2023-11-24T06:19:44.620224+0000","last_active":"2023-11-24T06:19:46.107052+0000","last_peered":"2023-11-24T06:19:46.107052+0000","last_clean":"2023-11-24T06:19:46.107052+0000","last_became_active":"2023-11-24T06:19:44.619946+0000","last_became_peered":"2023-11-24T06:19:44.619946+0000","last_unstale":"2023-11-24T06:19:46.107052+0000","last_undegraded":"2023-11-24T06:19:46.107052+0000","last_fullsized":"2023-11-24T06:19:46.107052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:34.002512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.994984+0000","last_change":"2023-11-24T06:19:43.605040+0000","last_active":"2023-11-24T06:19:44.994984+0000","last_peered":"2023-11-24T06:19:44.994984+0000","last_clean":"2023-11-24T06:19:44.994984+0000","last_became_active":"2023-11-24T06:19:43.604815+0000","last_became_peered":"2023-11-24T06:19:43.604815+0000","last_unstale":"2023-11-24T06:19:44.994984+0000","last_undegraded":"2023-11-24T06:19:44.994984+0000","last_fullsized":"2023-11-24T06:19:44.994984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:59.804014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144699+0000","last_change":"2023-11-24T06:19:44.605133+0000","last_active":"2023-11-24T06:19:46.144699+0000","last_peered":"2023-11-24T06:19:46.144699+0000","last_clean":"2023-11-24T06:19:46.144699+0000","last_became_active":"2023-11-24T06:19:44.604725+0000","last_became_peered":"2023-11-24T06:19:44.604725+0000","last_unstale":"2023-11-24T06:19:46.144699+0000","last_undegraded":"2023-11-24T06:19:46.144699+0000","last_fullsized":"2023-11-24T06:19:46.144699+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:06.278159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.610050+0000","last_change":"2023-11-24T06:19:43.599023+0000","last_active":"2023-11-24T06:19:44.610050+0000","last_peered":"2023-11-24T06:19:44.610050+0000","last_clean":"2023-11-24T06:19:44.610050+0000","last_became_active":"2023-11-24T06:19:43.598728+0000","last_became_peered":"2023-11-24T06:19:43.598728+0000","last_unstale":"2023-11-24T06:19:44.610050+0000","last_undegraded":"2023-11-24T06:19:44.610050+0000","last_fullsized":"2023-11-24T06:19:44.610050+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:30.429443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144510+0000","last_change":"2023-11-24T06:19:44.620299+0000","last_active":"2023-11-24T06:19:46.144510+0000","last_peered":"2023-11-24T06:19:46.144510+0000","last_clean":"2023-11-24T06:19:46.144510+0000","last_became_active":"2023-11-24T06:19:44.619966+0000","last_became_peered":"2023-11-24T06:19:44.619966+0000","last_unstale":"2023-11-24T06:19:46.144510+0000","last_undegraded":"2023-11-24T06:19:46.144510+0000","last_fullsized":"2023-11-24T06:19:46.144510+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:25:57.248982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624436+0000","last_change":"2023-11-24T06:19:43.607279+0000","last_active":"2023-11-24T06:19:46.624436+0000","last_peered":"2023-11-24T06:19:46.624436+0000","last_clean":"2023-11-24T06:19:46.624436+0000","last_became_active":"2023-11-24T06:19:43.607024+0000","last_became_peered":"2023-11-24T06:19:43.607024+0000","last_unstale":"2023-11-24T06:19:46.624436+0000","last_undegraded":"2023-11-24T06:19:46.624436+0000","last_fullsized":"2023-11-24T06:19:46.624436+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:30.429443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144583+0000","last_change":"2023-11-24T06:19:44.615777+0000","last_active":"2023-11-24T06:19:46.144583+0000","last_peered":"2023-11-24T06:19:46.144583+0000","last_clean":"2023-11-24T06:19:46.144583+0000","last_became_active":"2023-11-24T06:19:44.615495+0000","last_became_peered":"2023-11-24T06:19:44.615495+0000","last_unstale":"2023-11-24T06:19:46.144583+0000","last_undegraded":"2023-11-24T06:19:46.144583+0000","last_fullsized":"2023-11-24T06:19:46.144583+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:07.005490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.610512+0000","last_change":"2023-11-24T06:19:43.606633+0000","last_active":"2023-11-24T06:19:44.610512+0000","last_peered":"2023-11-24T06:19:44.610512+0000","last_clean":"2023-11-24T06:19:44.610512+0000","last_became_active":"2023-11-24T06:19:43.606347+0000","last_became_peered":"2023-11-24T06:19:43.606347+0000","last_unstale":"2023-11-24T06:19:44.610512+0000","last_undegraded":"2023-11-24T06:19:44.610512+0000","last_fullsized":"2023-11-24T06:19:44.610512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:54.775345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144487+0000","last_change":"2023-11-24T06:19:44.615445+0000","last_active":"2023-11-24T06:19:46.144487+0000","last_peered":"2023-11-24T06:19:46.144487+0000","last_clean":"2023-11-24T06:19:46.144487+0000","last_became_active":"2023-11-24T06:19:44.615034+0000","last_became_peered":"2023-11-24T06:19:44.615034+0000","last_unstale":"2023-11-24T06:19:46.144487+0000","last_undegraded":"2023-11-24T06:19:46.144487+0000","last_fullsized":"2023-11-24T06:19:46.144487+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:01.710358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.609592+0000","last_change":"2023-11-24T06:19:43.606153+0000","last_active":"2023-11-24T06:19:44.609592+0000","last_peered":"2023-11-24T06:19:44.609592+0000","last_clean":"2023-11-24T06:19:44.609592+0000","last_became_active":"2023-11-24T06:19:43.605891+0000","last_became_peered":"2023-11-24T06:19:43.605891+0000","last_unstale":"2023-11-24T06:19:44.609592+0000","last_undegraded":"2023-11-24T06:19:44.609592+0000","last_fullsized":"2023-11-24T06:19:44.609592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:54.775345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603558+0000","last_change":"2023-11-24T06:19:44.615573+0000","last_active":"2023-11-24T06:19:45.603558+0000","last_peered":"2023-11-24T06:19:45.603558+0000","last_clean":"2023-11-24T06:19:45.603558+0000","last_became_active":"2023-11-24T06:19:44.615282+0000","last_became_peered":"2023-11-24T06:19:44.615282+0000","last_unstale":"2023-11-24T06:19:45.603558+0000","last_undegraded":"2023-11-24T06:19:45.603558+0000","last_fullsized":"2023-11-24T06:19:45.603558+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:03:17.543360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603030+0000","last_change":"2023-11-24T06:19:43.602857+0000","last_active":"2023-11-24T06:19:45.603030+0000","last_peered":"2023-11-24T06:19:45.603030+0000","last_clean":"2023-11-24T06:19:45.603030+0000","last_became_active":"2023-11-24T06:19:43.602545+0000","last_became_peered":"2023-11-24T06:19:43.602545+0000","last_unstale":"2023-11-24T06:19:45.603030+0000","last_undegraded":"2023-11-24T06:19:45.603030+0000","last_fullsized":"2023-11-24T06:19:45.603030+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:30.429443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.144586+0000","last_change":"2023-11-24T06:19:44.623291+0000","last_active":"2023-11-24T06:19:46.144586+0000","last_peered":"2023-11-24T06:19:46.144586+0000","last_clean":"2023-11-24T06:19:46.144586+0000","last_became_active":"2023-11-24T06:19:44.622972+0000","last_became_peered":"2023-11-24T06:19:44.622972+0000","last_unstale":"2023-11-24T06:19:46.144586+0000","last_undegraded":"2023-11-24T06:19:46.144586+0000","last_fullsized":"2023-11-24T06:19:46.144586+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:55.783521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107458+0000","last_change":"2023-11-24T06:19:43.608298+0000","last_active":"2023-11-24T06:19:46.107458+0000","last_peered":"2023-11-24T06:19:46.107458+0000","last_clean":"2023-11-24T06:19:46.107458+0000","last_became_active":"2023-11-24T06:19:43.606141+0000","last_became_peered":"2023-11-24T06:19:43.606141+0000","last_unstale":"2023-11-24T06:19:46.107458+0000","last_undegraded":"2023-11-24T06:19:46.107458+0000","last_fullsized":"2023-11-24T06:19:46.107458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:36:45.636780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604155+0000","last_change":"2023-11-24T06:19:44.618409+0000","last_active":"2023-11-24T06:19:45.604155+0000","last_peered":"2023-11-24T06:19:45.604155+0000","last_clean":"2023-11-24T06:19:45.604155+0000","last_became_active":"2023-11-24T06:19:44.618085+0000","last_became_peered":"2023-11-24T06:19:44.618085+0000","last_unstale":"2023-11-24T06:19:45.604155+0000","last_undegraded":"2023-11-24T06:19:45.604155+0000","last_fullsized":"2023-11-24T06:19:45.604155+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:34.436870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.625760+0000","last_change":"2023-11-24T06:19:43.606647+0000","last_active":"2023-11-24T06:19:46.625760+0000","last_peered":"2023-11-24T06:19:46.625760+0000","last_clean":"2023-11-24T06:19:46.625760+0000","last_became_active":"2023-11-24T06:19:43.606298+0000","last_became_peered":"2023-11-24T06:19:43.606298+0000","last_unstale":"2023-11-24T06:19:46.625760+0000","last_undegraded":"2023-11-24T06:19:46.625760+0000","last_fullsized":"2023-11-24T06:19:46.625760+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:05.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624350+0000","last_change":"2023-11-24T06:19:44.612790+0000","last_active":"2023-11-24T06:19:46.624350+0000","last_peered":"2023-11-24T06:19:46.624350+0000","last_clean":"2023-11-24T06:19:46.624350+0000","last_became_active":"2023-11-24T06:19:44.612566+0000","last_became_peered":"2023-11-24T06:19:44.612566+0000","last_unstale":"2023-11-24T06:19:46.624350+0000","last_undegraded":"2023-11-24T06:19:46.624350+0000","last_fullsized":"2023-11-24T06:19:46.624350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:28:29.540648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.107102+0000","last_change":"2023-11-24T06:19:43.606597+0000","last_active":"2023-11-24T06:19:46.107102+0000","last_peered":"2023-11-24T06:19:46.107102+0000","last_clean":"2023-11-24T06:19:46.107102+0000","last_became_active":"2023-11-24T06:19:43.606174+0000","last_became_peered":"2023-11-24T06:19:43.606174+0000","last_unstale":"2023-11-24T06:19:46.107102+0000","last_undegraded":"2023-11-24T06:19:46.107102+0000","last_fullsized":"2023-11-24T06:19:46.107102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:59.804014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.622556+0000","last_change":"2023-11-24T06:19:44.617331+0000","last_active":"2023-11-24T06:19:44.622556+0000","last_peered":"2023-11-24T06:19:44.622556+0000","last_clean":"2023-11-24T06:19:44.622556+0000","last_became_active":"2023-11-24T06:19:44.616977+0000","last_became_peered":"2023-11-24T06:19:44.616977+0000","last_unstale":"2023-11-24T06:19:44.622556+0000","last_undegraded":"2023-11-24T06:19:44.622556+0000","last_fullsized":"2023-11-24T06:19:44.622556+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:43:03.428940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624390+0000","last_change":"2023-11-24T06:19:43.603879+0000","last_active":"2023-11-24T06:19:46.624390+0000","last_peered":"2023-11-24T06:19:46.624390+0000","last_clean":"2023-11-24T06:19:46.624390+0000","last_became_active":"2023-11-24T06:19:43.603611+0000","last_became_peered":"2023-11-24T06:19:43.603611+0000","last_unstale":"2023-11-24T06:19:46.624390+0000","last_undegraded":"2023-11-24T06:19:46.624390+0000","last_fullsized":"2023-11-24T06:19:46.624390+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:59.804014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.624492+0000","last_change":"2023-11-24T06:19:44.618178+0000","last_active":"2023-11-24T06:19:46.624492+0000","last_peered":"2023-11-24T06:19:46.624492+0000","last_clean":"2023-11-24T06:19:46.624492+0000","last_became_active":"2023-11-24T06:19:44.617903+0000","last_became_peered":"2023-11-24T06:19:44.617903+0000","last_unstale":"2023-11-24T06:19:46.624492+0000","last_undegraded":"2023-11-24T06:19:46.624492+0000","last_fullsized":"2023-11-24T06:19:46.624492+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:45:21.055969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:46.106926+0000","last_change":"2023-11-24T06:19:43.604070+0000","last_active":"2023-11-24T06:19:46.106926+0000","last_peered":"2023-11-24T06:19:46.106926+0000","last_clean":"2023-11-24T06:19:46.106926+0000","last_became_active":"2023-11-24T06:19:43.603750+0000","last_became_peered":"2023-11-24T06:19:43.603750+0000","last_unstale":"2023-11-24T06:19:46.106926+0000","last_undegraded":"2023-11-24T06:19:46.106926+0000","last_fullsized":"2023-11-24T06:19:46.106926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:36:56.366458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.604484+0000","last_change":"2023-11-24T06:19:44.609460+0000","last_active":"2023-11-24T06:19:45.604484+0000","last_peered":"2023-11-24T06:19:45.604484+0000","last_clean":"2023-11-24T06:19:45.604484+0000","last_became_active":"2023-11-24T06:19:44.609125+0000","last_became_peered":"2023-11-24T06:19:44.609125+0000","last_unstale":"2023-11-24T06:19:45.604484+0000","last_undegraded":"2023-11-24T06:19:45.604484+0000","last_fullsized":"2023-11-24T06:19:45.604484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:35:04.205238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.609699+0000","last_change":"2023-11-24T06:19:43.603124+0000","last_active":"2023-11-24T06:19:44.609699+0000","last_peered":"2023-11-24T06:19:44.609699+0000","last_clean":"2023-11-24T06:19:44.609699+0000","last_became_active":"2023-11-24T06:19:43.602856+0000","last_became_peered":"2023-11-24T06:19:43.602856+0000","last_unstale":"2023-11-24T06:19:44.609699+0000","last_undegraded":"2023-11-24T06:19:44.609699+0000","last_fullsized":"2023-11-24T06:19:44.609699+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:05.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.622114+0000","last_change":"2023-11-24T06:19:44.615975+0000","last_active":"2023-11-24T06:19:44.622114+0000","last_peered":"2023-11-24T06:19:44.622114+0000","last_clean":"2023-11-24T06:19:44.622114+0000","last_became_active":"2023-11-24T06:19:44.615687+0000","last_became_peered":"2023-11-24T06:19:44.615687+0000","last_unstale":"2023-11-24T06:19:44.622114+0000","last_undegraded":"2023-11-24T06:19:44.622114+0000","last_fullsized":"2023-11-24T06:19:44.622114+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:45.302016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.605071+0000","last_change":"2023-11-24T06:19:43.603830+0000","last_active":"2023-11-24T06:19:45.605071+0000","last_peered":"2023-11-24T06:19:45.605071+0000","last_clean":"2023-11-24T06:19:45.605071+0000","last_became_active":"2023-11-24T06:19:43.602570+0000","last_became_peered":"2023-11-24T06:19:43.602570+0000","last_unstale":"2023-11-24T06:19:45.605071+0000","last_undegraded":"2023-11-24T06:19:45.605071+0000","last_fullsized":"2023-11-24T06:19:45.605071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:58:49.304457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:19:45.603934+0000","last_change":"2023-11-24T06:19:44.606372+0000","last_active":"2023-11-24T06:19:45.603934+0000","last_peered":"2023-11-24T06:19:45.603934+0000","last_clean":"2023-11-24T06:19:45.603934+0000","last_became_active":"2023-11-24T06:19:44.606069+0000","last_became_peered":"2023-11-24T06:19:44.606069+0000","last_unstale":"2023-11-24T06:19:45.603934+0000","last_undegraded":"2023-11-24T06:19:45.603934+0000","last_fullsized":"2023-11-24T06:19:45.603934+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:43.582109+0000","last_clean_scrub_stamp":"2023-11-24T06:19:43.582109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:58:50.312633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.609826+0000","last_change":"2023-11-24T06:19:43.607414+0000","last_active":"2023-11-24T06:19:44.609826+0000","last_peered":"2023-11-24T06:19:44.609826+0000","last_clean":"2023-11-24T06:19:44.609826+0000","last_became_active":"2023-11-24T06:19:43.607080+0000","last_became_peered":"2023-11-24T06:19:43.607080+0000","last_unstale":"2023-11-24T06:19:44.609826+0000","last_undegraded":"2023-11-24T06:19:44.609826+0000","last_fullsized":"2023-11-24T06:19:44.609826+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:19:42.573934+0000","last_clean_scrub_stamp":"2023-11-24T06:19:42.573934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:00.702182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:19:44.600597+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:19:44.600597+0000","last_peered":"2023-11-24T06:19:44.600597+0000","last_clean":"2023-11-24T06:19:44.600597+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:19:44.600597+0000","last_undegraded":"2023-11-24T06:19:44.600597+0000","last_fullsized":"2023-11-24T06:19:44.600597+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":741583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.64600000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.53600000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.502}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.61699999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.58399999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.67500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":145441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.59399999999999997}]},{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":6.423,"5min":6.423,"15min":6.423},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.44400000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":12.554,"5min":12.554,"15min":12.554},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.65800000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.621}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":152759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.495}]},{"osd":3,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.62},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.59799999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.746}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":735193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.26600000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.31},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.34300000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.433}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.54100000000000004}]},{"osd":6,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.40200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":151195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.23899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.372}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.32400000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.35299999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.44800000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.39200000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":152759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.246},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.378}]},{"osd":1,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.31},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.36299999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.435}]},{"osd":7,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":144929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.496},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.64900000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.63100000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.374},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.40600000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59399999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":151797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.04,"5min":3.04,"15min":3.04},"last":0.443}]},{"osd":1,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.60799999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":10.198,"5min":10.198,"15min":10.198},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":10.215999999999999,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64100000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.628}]},{"osd":7,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.69399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:19:49.487 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T06:19:49.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T06:19:49.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.650+0000 7fc98e242700 1 Processor -- start 2023-11-24T06:19:49.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.650+0000 7fc98e242700 1 -- start start 2023-11-24T06:19:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc98e242700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc98815e150 0x7fc98806f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc98e242700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9880b3bf0 0x7fc98806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc98e242700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc988073f90 0x7fc988074440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc98e242700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc988074980 con 0x7fc9880b3bf0 2023-11-24T06:19:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc98e242700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9880b4360 con 0x7fc988073f90 2023-11-24T06:19:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc98e242700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98806d8d0 con 0x7fc98815e150 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9880b3bf0 0x7fc98806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9880b3bf0 0x7fc98806e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36968/0 (socket says 172.21.15.89:36968) 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 -- 172.21.15.89:0/1208992312 learned_addr learned my addr 172.21.15.89:0/1208992312 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc987fff700 1 --2- 172.21.15.89:0/1208992312 >> v2:172.21.15.123:3300/0 conn(0x7fc988073f90 0x7fc988074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc9877fe700 1 --2- 172.21.15.89:0/1208992312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc98815e150 0x7fc98806f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 -- 172.21.15.89:0/1208992312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc98815e150 msgr2=0x7fc98806f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 --2- 172.21.15.89:0/1208992312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc98815e150 0x7fc98806f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 -- 172.21.15.89:0/1208992312 >> v2:172.21.15.123:3300/0 conn(0x7fc988073f90 msgr2=0x7fc988074440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 --2- 172.21.15.89:0/1208992312 >> v2:172.21.15.123:3300/0 conn(0x7fc988073f90 0x7fc988074440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 -- 172.21.15.89:0/1208992312 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9880b0590 con 0x7fc9880b3bf0 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc987fff700 1 --2- 172.21.15.89:0/1208992312 >> v2:172.21.15.123:3300/0 conn(0x7fc988073f90 0x7fc988074440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc986ffd700 1 --2- 172.21.15.89:0/1208992312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9880b3bf0 0x7fc98806e5c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc9880d1a10 tx=0x7fc97c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421b1168f5bccc1c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.651+0000 7fc9867fc700 1 -- 172.21.15.89:0/1208992312 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00b870 con 0x7fc9880b3bf0 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc9867fc700 1 -- 172.21.15.89:0/1208992312 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c00b9d0 con 0x7fc9880b3bf0 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc9867fc700 1 -- 172.21.15.89:0/1208992312 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00bd60 con 0x7fc9880b3bf0 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 -- 172.21.15.89:0/1208992312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9880b3bf0 msgr2=0x7fc98806e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 --2- 172.21.15.89:0/1208992312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9880b3bf0 0x7fc98806e5c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc9880d1a10 tx=0x7fc97c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 -- 172.21.15.89:0/1208992312 shutdown_connections 2023-11-24T06:19:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 --2- 172.21.15.89:0/1208992312 >> v2:172.21.15.123:3300/0 conn(0x7fc988073f90 0x7fc988074440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 --2- 172.21.15.89:0/1208992312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc98815e150 0x7fc98806f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 --2- 172.21.15.89:0/1208992312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9880b3bf0 0x7fc98806e5c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 -- 172.21.15.89:0/1208992312 >> 172.21.15.89:0/1208992312 conn(0x7fc9880d0be0 msgr2=0x7fc9880d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 -- 172.21.15.89:0/1208992312 shutdown_connections 2023-11-24T06:19:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.652+0000 7fc98e242700 1 -- 172.21.15.89:0/1208992312 wait complete. 2023-11-24T06:19:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.653+0000 7fc98e242700 1 Processor -- start 2023-11-24T06:19:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.653+0000 7fc98e242700 1 -- start start 2023-11-24T06:19:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc98e242700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc988073f90 0x7fc988155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc98e242700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98815e150 0x7fc988152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc98e242700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880b1770 0x7fc9880b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc98e242700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880b2140 con 0x7fc98815e150 2023-11-24T06:19:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc98e242700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc988154170 con 0x7fc9880b1770 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc98e242700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc988154450 con 0x7fc988073f90 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc986ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98815e150 0x7fc988152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc986ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98815e150 0x7fc988152780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36974/0 (socket says 172.21.15.89:36974) 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc986ffd700 1 -- 172.21.15.89:0/1421744835 learned_addr learned my addr 172.21.15.89:0/1421744835 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc9877fe700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc988073f90 0x7fc988155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.654+0000 7fc987fff700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880b1770 0x7fc9880b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc986ffd700 1 -- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc988073f90 msgr2=0x7fc988155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc986ffd700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc988073f90 0x7fc988155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc986ffd700 1 -- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880b1770 msgr2=0x7fc9880b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc986ffd700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880b1770 0x7fc9880b1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc986ffd700 1 -- 172.21.15.89:0/1421744835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c009550 con 0x7fc98815e150 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc9877fe700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc988073f90 0x7fc988155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc987fff700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880b1770 0x7fc9880b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc986ffd700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98815e150 0x7fc988152780 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc97c000c00 tx=0x7fc97c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc96ffff700 1 -- 172.21.15.89:0/1421744835 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c004540 con 0x7fc98815e150 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc96ffff700 1 -- 172.21.15.89:0/1421744835 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c0036a0 con 0x7fc98815e150 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc96ffff700 1 -- 172.21.15.89:0/1421744835 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c003a50 con 0x7fc98815e150 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc988154730 con 0x7fc98815e150 2023-11-24T06:19:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.655+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc988154cf0 con 0x7fc98815e150 2023-11-24T06:19:49.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.656+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc988154eb0 con 0x7fc98815e150 2023-11-24T06:19:49.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.656+0000 7fc96ffff700 1 -- 172.21.15.89:0/1421744835 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc97c021070 con 0x7fc98815e150 2023-11-24T06:19:49.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.657+0000 7fc96ffff700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc97406aed0 0x7fc97406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:49.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.657+0000 7fc96ffff700 1 -- 172.21.15.89:0/1421744835 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc97c08ac90 con 0x7fc98815e150 2023-11-24T06:19:49.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.657+0000 7fc9877fe700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc97406aed0 0x7fc97406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:49.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.657+0000 7fc9877fe700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc97406aed0 0x7fc97406d390 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc97800ba70 tx=0x7fc978008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:49.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.661+0000 7fc96ffff700 1 -- 172.21.15.89:0/1421744835 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc97c055dd0 con 0x7fc98815e150 2023-11-24T06:19:49.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.930+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc988068fb0 con 0x7fc98815e150 2023-11-24T06:19:49.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.932+0000 7fc96ffff700 1 -- 172.21.15.89:0/1421744835 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc97c0290e0 con 0x7fc98815e150 2023-11-24T06:19:49.935 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2023-11-24T06:19:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlwIfyKhAAMZcwOQGAm+XcH+mUKcTiUg== 2023-11-24T06:19:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: caps mds = "allow" 2023-11-24T06:19:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "allow r" 2023-11-24T06:19:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T06:19:49.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc97406aed0 msgr2=0x7fc97406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc97406aed0 0x7fc97406d390 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc97800ba70 tx=0x7fc978008040 comp rx=0 tx=0).stop 2023-11-24T06:19:49.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98815e150 msgr2=0x7fc988152780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:49.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98815e150 0x7fc988152780 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc97c000c00 tx=0x7fc97c004750 comp rx=0 tx=0).stop 2023-11-24T06:19:49.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 shutdown_connections 2023-11-24T06:19:49.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc97406aed0 0x7fc97406d390 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880b1770 0x7fc9880b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc988073f90 0x7fc988155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 --2- 172.21.15.89:0/1421744835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98815e150 0x7fc988152780 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:49.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 >> 172.21.15.89:0/1421744835 conn(0x7fc9880d0be0 msgr2=0x7fc98806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:49.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 shutdown_connections 2023-11-24T06:19:49.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:49.934+0000 7fc98e242700 1 -- 172.21.15.89:0/1421744835 wait complete. 2023-11-24T06:19:49.940 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2023-11-24T06:19:49.949 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:19:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.068+0000 7fe9e1431700 1 Processor -- start 2023-11-24T06:19:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.068+0000 7fe9e1431700 1 -- start start 2023-11-24T06:19:50.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.068+0000 7fe9e1431700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9dc15e480 0x7fe9dc0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.068+0000 7fe9e1431700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe9dc0d3610 0x7fe9dc0b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.068+0000 7fe9e1431700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc0b62c0 con 0x7fe9dc0b5e90 2023-11-24T06:19:50.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.068+0000 7fe9e1431700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9dc0d2a70 con 0x7fe9dc15e480 2023-11-24T06:19:50.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.068+0000 7fe9e1431700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9dc0d2d50 con 0x7fe9dc0d3610 2023-11-24T06:19:50.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.068+0000 7fe9daffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9dc15e480 0x7fe9dc0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9daffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9dc15e480 0x7fe9dc0d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36980/0 (socket says 172.21.15.89:36980) 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9daffd700 1 -- 172.21.15.89:0/2385844361 learned_addr learned my addr 172.21.15.89:0/2385844361 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9da7fc700 1 --2- 172.21.15.89:0/2385844361 >> v2:172.21.15.123:3300/0 conn(0x7fe9dc0d3610 0x7fe9dc0b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9daffd700 1 -- 172.21.15.89:0/2385844361 >> v2:172.21.15.123:3300/0 conn(0x7fe9dc0d3610 msgr2=0x7fe9dc0b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9daffd700 1 --2- 172.21.15.89:0/2385844361 >> v2:172.21.15.123:3300/0 conn(0x7fe9dc0d3610 0x7fe9dc0b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9daffd700 1 -- 172.21.15.89:0/2385844361 >> v1:172.21.15.123:6789/0 conn(0x7fe9dc0b5e90 legacy=0x7fe9dc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9daffd700 1 -- 172.21.15.89:0/2385844361 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0d3030 con 0x7fe9dc15e480 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9daffd700 1 --2- 172.21.15.89:0/2385844361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc0d2100 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc08c8e0 tx=0x7fe9d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d439859ac808819 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9d9ffb700 1 -- 172.21.15.89:0/2385844361 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000b870 con 0x7fe9dc15e480 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9d9ffb700 1 -- 172.21.15.89:0/2385844361 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d000b9d0 con 0x7fe9dc15e480 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.069+0000 7fe9d9ffb700 1 -- 172.21.15.89:0/2385844361 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000bd60 con 0x7fe9dc15e480 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.070+0000 7fe9e1431700 1 -- 172.21.15.89:0/2385844361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 msgr2=0x7fe9dc0d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.070+0000 7fe9e1431700 1 --2- 172.21.15.89:0/2385844361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc0d2100 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc08c8e0 tx=0x7fe9d0009bd0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.070+0000 7fe9e1431700 1 -- 172.21.15.89:0/2385844361 shutdown_connections 2023-11-24T06:19:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.070+0000 7fe9e1431700 1 --2- 172.21.15.89:0/2385844361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc0d2100 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.070+0000 7fe9e1431700 1 --2- 172.21.15.89:0/2385844361 >> v2:172.21.15.123:3300/0 conn(0x7fe9dc0d3610 0x7fe9dc0b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.070+0000 7fe9e1431700 1 -- 172.21.15.89:0/2385844361 >> 172.21.15.89:0/2385844361 conn(0x7fe9dc0d0950 msgr2=0x7fe9dc0d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.070+0000 7fe9e1431700 1 -- 172.21.15.89:0/2385844361 shutdown_connections 2023-11-24T06:19:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.070+0000 7fe9e1431700 1 -- 172.21.15.89:0/2385844361 wait complete. 2023-11-24T06:19:50.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.071+0000 7fe9e1431700 1 Processor -- start 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.071+0000 7fe9e1431700 1 -- start start 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9e1431700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9dc0b5e90 0x7fe9dc0b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9e1431700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9dc0d3610 0x7fe9dc0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9e1431700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc15e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9e1431700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0b59c0 con 0x7fe9dc15e480 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9e1431700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0cddf0 con 0x7fe9dc0d3610 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9e1431700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc14b250 con 0x7fe9dc0b5e90 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc15e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36990/0 (socket says 172.21.15.89:36990) 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 -- 172.21.15.89:0/3413205196 learned_addr learned my addr 172.21.15.89:0/3413205196 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9da7fc700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9dc0d3610 0x7fe9dc0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9daffd700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9dc0b5e90 0x7fe9dc0b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 -- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9dc0b5e90 msgr2=0x7fe9dc0b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9dc0b5e90 0x7fe9dc0b5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 -- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9dc0d3610 msgr2=0x7fe9dc0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9dc0d3610 0x7fe9dc0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 -- 172.21.15.89:0/3413205196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d0009550 con 0x7fe9dc15e480 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9daffd700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9dc0b5e90 0x7fe9dc0b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9db7fe700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc15e860 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400cb40 tx=0x7fe9d400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9c37fe700 1 -- 172.21.15.89:0/3413205196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4003c00 con 0x7fe9dc15e480 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9c37fe700 1 -- 172.21.15.89:0/3413205196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d4003d60 con 0x7fe9dc15e480 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9c37fe700 1 -- 172.21.15.89:0/3413205196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d400ab90 con 0x7fe9dc15e480 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.072+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9dc14b4f0 con 0x7fe9dc15e480 2023-11-24T06:19:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.073+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9dc14bab0 con 0x7fe9dc15e480 2023-11-24T06:19:50.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.073+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9dc0ce240 con 0x7fe9dc15e480 2023-11-24T06:19:50.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.074+0000 7fe9c37fe700 1 -- 172.21.15.89:0/3413205196 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe9d400acf0 con 0x7fe9dc15e480 2023-11-24T06:19:50.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.074+0000 7fe9c37fe700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9c806aed0 0x7fe9c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.074+0000 7fe9c37fe700 1 -- 172.21.15.89:0/3413205196 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9d40895e0 con 0x7fe9dc15e480 2023-11-24T06:19:50.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.074+0000 7fe9daffd700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9c806aed0 0x7fe9c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.075+0000 7fe9daffd700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9c806aed0 0x7fe9c806d390 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0caae0 tx=0x7fe9d0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.079+0000 7fe9c37fe700 1 -- 172.21.15.89:0/3413205196 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9d4054720 con 0x7fe9dc15e480 2023-11-24T06:19:50.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.313+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9dc0751e0 con 0x7fe9dc15e480 2023-11-24T06:19:50.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.313+0000 7fe9c37fe700 1 -- 172.21.15.89:0/3413205196 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fe9d4057d40 con 0x7fe9dc15e480 2023-11-24T06:19:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:50.317 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/503786989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":503786989},{"type":"v1","addr":"172.21.15.123:6834","nonce":503786989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3511736853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3511736853},{"type":"v1","addr":"172.21.15.89:6836","nonce":3511736853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5091,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053513777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2053513777},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053513777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:19:43.989850+0000","modified":"2023-11-24T06:19:47.380732+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6835/1601724762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1601724762},{"type":"v1","addr":"172.21.15.123:6835","nonce":1601724762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:19:50.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9c806aed0 msgr2=0x7fe9c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9c806aed0 0x7fe9c806d390 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0caae0 tx=0x7fe9d0008030 comp rx=0 tx=0).stop 2023-11-24T06:19:50.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 msgr2=0x7fe9dc15e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc15e860 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400cb40 tx=0x7fe9d400cf00 comp rx=0 tx=0).stop 2023-11-24T06:19:50.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 shutdown_connections 2023-11-24T06:19:50.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc15e480 0x7fe9dc15e860 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9dc0b5e90 0x7fe9dc0b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9c806aed0 0x7fe9c806d390 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 --2- 172.21.15.89:0/3413205196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9dc0d3610 0x7fe9dc0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 >> 172.21.15.89:0/3413205196 conn(0x7fe9dc0d0950 msgr2=0x7fe9dc0b9280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:50.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 shutdown_connections 2023-11-24T06:19:50.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.315+0000 7fe9e1431700 1 -- 172.21.15.89:0/3413205196 wait complete. 2023-11-24T06:19:50.322 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2023-11-24T06:19:50.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T06:19:43.989850+0000', 'modified': '2023-11-24T06:19:47.380732+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5103}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5103': {'gid': 5103, 'name': 'b', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6835/1601724762', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6833', 'nonce': 1601724762}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 1601724762}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T06:19:50.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T06:19:50.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-24T06:19:50.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.858+0000 7f3ed9cdc700 1 Processor -- start 2023-11-24T06:19:50.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.858+0000 7f3ed9cdc700 1 -- start start 2023-11-24T06:19:50.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.858+0000 7f3ed9cdc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ed40b1a70 0x7f3ed40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.858+0000 7f3ed9cdc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed40cfd40 con 0x7f3ed415ae20 2023-11-24T06:19:50.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.858+0000 7f3ed9cdc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed40affa0 con 0x7f3ed415b200 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.858+0000 7f3ed9cdc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed40d0020 con 0x7f3ed40b1a70 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed37fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3ed415ae20 0x7f3ed40b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40498/0 (socket says 172.21.15.89:40498) 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed37fe700 1 -- 172.21.15.89:0/3821684868 learned_addr learned my addr 172.21.15.89:0/3821684868 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ed40b1a70 0x7f3ed40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502291093 0 0) 0x7f3ed40cfd40 con 0x7f3ed415ae20 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc0035b0 con 0x7f3ed415ae20 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236211071 0 0) 0x7f3ebc0035b0 con 0x7f3ed415ae20 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ed40b1a70 msgr2=0x7f3ed40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ed40b1a70 0x7f3ed40b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 >> v1:172.21.15.123:6789/0 conn(0x7f3ed415b200 legacy=0x7f3ed4162b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed40b0280 con 0x7f3ed415ae20 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f3ec4001670 con 0x7f3ed415ae20 2023-11-24T06:19:50.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 >> v1:172.21.15.89:6789/0 conn(0x7f3ed415ae20 legacy=0x7f3ed40b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ebc003a30 0x7f3ebc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ebc004850 0x7f3ebc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ed40b1a70 0x7f3ebc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed40d0020 con 0x7f3ebc004850 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00bac0 con 0x7f3ebc003a30 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00bc80 con 0x7f3ed40b1a70 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ebc004850 0x7f3ebc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ebc003a30 0x7f3ebc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.859+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ed40b1a70 0x7f3ebc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ed40b1a70 msgr2=0x7f3ebc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ed40b1a70 0x7f3ebc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ebc003a30 msgr2=0x7f3ebc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ebc003a30 0x7f3ebc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/3821684868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc0035b0 con 0x7f3ebc004850 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ebc004850 0x7f3ebc004cd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8007fd0 tx=0x7f3ebc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f872e3921bad4f2f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80052d0 con 0x7f3ebc004850 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec8005430 con 0x7f3ebc004850 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed27fc700 1 -- 172.21.15.89:0/3821684868 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80038c0 con 0x7f3ebc004850 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/3821684868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ebc004850 msgr2=0x7f3ebc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed9cdc700 1 --2- 172.21.15.89:0/3821684868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ebc004850 0x7f3ebc004cd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8007fd0 tx=0x7f3ebc005880 comp rx=0 tx=0).stop 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed3fff700 1 -- 172.21.15.89:0/3821684868 reap_dead start 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/3821684868 shutdown_connections 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/3821684868 >> 172.21.15.89:0/3821684868 conn(0x7f3ed40b57b0 msgr2=0x7f3ed40b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:50.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/3821684868 shutdown_connections 2023-11-24T06:19:50.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.860+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/3821684868 wait complete. 2023-11-24T06:19:50.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.862+0000 7f3ed9cdc700 1 Processor -- start 2023-11-24T06:19:50.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.862+0000 7f3ed9cdc700 1 -- start start 2023-11-24T06:19:50.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.862+0000 7f3ed9cdc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ebc004850 0x7f3ed415caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.862+0000 7f3ed9cdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ed415e140 0x7f3ed40cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.862+0000 7f3ed9cdc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed414b5d0 0x7f3ed40b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed9cdc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed415dd20 con 0x7f3ed415e140 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed9cdc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed415d0a0 con 0x7f3ed414b5d0 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed9cdc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed415d380 con 0x7f3ebc004850 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ed415e140 0x7f3ed40cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ed415e140 0x7f3ed40cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37008/0 (socket says 172.21.15.89:37008) 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/1458019424 learned_addr learned my addr 172.21.15.89:0/1458019424 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed414b5d0 0x7f3ed40b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ebc004850 0x7f3ed415caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ebc004850 msgr2=0x7f3ed415caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ebc004850 0x7f3ed415caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed414b5d0 msgr2=0x7f3ed40b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed414b5d0 0x7f3ed40b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/1458019424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed415d660 con 0x7f3ed415e140 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed414b5d0 0x7f3ed40b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ebc004850 0x7f3ed415caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ed415e140 0x7f3ed40cc030 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3ed406d1d0 tx=0x7f3ec8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ebbfff700 1 -- 172.21.15.89:0/1458019424 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8007240 con 0x7f3ed415e140 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ebbfff700 1 -- 172.21.15.89:0/1458019424 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec80073a0 con 0x7f3ed415e140 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/1458019424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed415d940 con 0x7f3ed415e140 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ebbfff700 1 -- 172.21.15.89:0/1458019424 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8007730 con 0x7f3ed415e140 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.863+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/1458019424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed414ba50 con 0x7f3ed415e140 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.865+0000 7f3ebbfff700 1 -- 172.21.15.89:0/1458019424 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3ec8003680 con 0x7f3ed415e140 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.865+0000 7f3ebbfff700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3ec006ae00 0x7f3ec006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.865+0000 7f3ebbfff700 1 -- 172.21.15.89:0/1458019424 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec808a570 con 0x7f3ed415e140 2023-11-24T06:19:50.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.866+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3ec006ae00 0x7f3ec006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.866+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3ec006ae00 0x7f3ec006d2c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4007e10 tx=0x7f3ec4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.867+0000 7f3ee5ed5b80 1 Processor -- start 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.867+0000 7f3ee5ed5b80 1 -- start start 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02dd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d02ddd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 0x5607d02c0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d020a770 con 0x5607d0297080 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d01f1750 con 0x5607d02aced0 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607d01ef070 con 0x5607d0294490 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ed37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02dd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ed37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02dd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37014/0 (socket says 172.21.15.89:37014) 2023-11-24T06:19:50.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ed37fe700 1 -- 172.21.15.89:0/1604310787 learned_addr learned my addr 172.21.15.89:0/1604310787 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d02ddd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.868+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 0x5607d02c0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3ed37fe700 1 -- 172.21.15.89:0/1604310787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 msgr2=0x5607d02c0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 0x5607d02c0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3ed37fe700 1 -- 172.21.15.89:0/1604310787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 msgr2=0x5607d02ddd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d02ddd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3ed37fe700 1 -- 172.21.15.89:0/1604310787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607d00675e0 con 0x5607d0297080 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02dd840 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40072b0 tx=0x7f3ec4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49014ae0c3e5e69 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3eb97fa700 1 -- 172.21.15.89:0/1604310787 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec40157e0 con 0x5607d0297080 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3eb97fa700 1 -- 172.21.15.89:0/1604310787 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec4015940 con 0x5607d0297080 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.869+0000 7f3eb97fa700 1 -- 172.21.15.89:0/1604310787 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4015cb0 con 0x5607d0297080 2023-11-24T06:19:50.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/1604310787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 msgr2=0x5607d02dd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02dd840 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40072b0 tx=0x7f3ec4014000 comp rx=0 tx=0).stop 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/1604310787 shutdown_connections 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 0x5607d02c0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02dd840 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/1604310787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d02ddd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/1604310787 >> 172.21.15.89:0/1604310787 conn(0x5607d01c5310 msgr2=0x5607d02dbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/1604310787 shutdown_connections 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/1604310787 wait complete. 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 -- start start 2023-11-24T06:19:50.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.870+0000 7f3ee5ed5b80 1 Processor -- start 2023-11-24T06:19:50.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0294490 0x5607d02c4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02d9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d02aced0 0x5607d02d3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d01ea960 con 0x5607d0297080 2023-11-24T06:19:50.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d01a4510 con 0x5607d0294490 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607d01a41b0 con 0x5607d02aced0 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ee5ed5b80 10 client.0 _ll_register_callbacks cb 0x5607d00cfa30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ed2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02d9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ed2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02d9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37022/0 (socket says 172.21.15.89:37022) 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/3818274235 learned_addr learned my addr 172.21.15.89:0/3818274235 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0294490 0x5607d02c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d02aced0 0x5607d02d3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d02aced0 msgr2=0x5607d02d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.871+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d02aced0 0x5607d02d3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0294490 msgr2=0x5607d02c4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0294490 0x5607d02c4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ed2ffd700 1 -- 172.21.15.89:0/3818274235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed40b0280 con 0x5607d0297080 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0294490 0x5607d02c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02d9100 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8088a30 tx=0x7f3ec8059440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8eb750870f6ff6c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3eb27fc700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3eb27fc700 1 -- 172.21.15.89:0/3818274235 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80595f0 con 0x5607d0297080 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3eb27fc700 1 -- 172.21.15.89:0/3818274235 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec8059750 con 0x5607d0297080 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/3818274235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607d0289a40 con 0x5607d0297080 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3eb27fc700 1 -- 172.21.15.89:0/3818274235 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ec8058880 con 0x5607d0297080 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 10 client.5277 fetch_fsmap learned FSMap version 17 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/3818274235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5607d0013d90 con 0x5607d0297080 2023-11-24T06:19:50.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3eb27fc700 1 -- 172.21.15.89:0/3818274235 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5607d0289a40 con 0x5607d0297080 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3eb27fc700 1 -- 172.21.15.89:0/3818274235 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3ec8061200 con 0x5607d0297080 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 10 client.5277 fetch_fsmap finished waiting for FSMap version 17 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 10 client.5277 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 20 client.5277 populate_metadata read hostname 'smithi089' 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] conn(0x5607d02d6930 0x5607d02ee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 4 client.5277 mds_command: new command op to 5103 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/3818274235 --> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607d0265840 con 0x5607d02d6930 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.872+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] conn(0x5607d02d6930 0x5607d02ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.873+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] conn(0x5607d02d6930 0x5607d02ee800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c84eead6490779e server_cookie=e91f3f35e064a8df in_seq=0 out_seq=0 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.873+0000 7f3eb27fc700 10 client.5277 ms_handle_connect on v2:172.21.15.123:6833/1601724762 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3eb27fc700 1 -- 172.21.15.89:0/3818274235 <== mds.0 v2:172.21.15.123:6833/1601724762 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5607d0265840 con 0x5607d02d6930 2023-11-24T06:19:50.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3eb27fc700 10 client.5277 handle_command_reply: tid=0 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 client.5277 shutdown 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/3818274235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 msgr2=0x5607d02d9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02d9100 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8088a30 tx=0x7f3ec8059440 comp rx=0 tx=0).stop 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/3818274235 shutdown_connections 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d02aced0 0x5607d02d3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] conn(0x5607d02d6930 0x5607d02ee800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0294490 0x5607d02c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/3818274235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0297080 0x5607d02d9100 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3eb27fc700 0 client.5277 ms_handle_reset on v2:172.21.15.123:6833/1601724762 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/3818274235 >> 172.21.15.89:0/3818274235 conn(0x5607d01c5310 msgr2=0x5607d02dfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:50.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/3818274235 shutdown_connections 2023-11-24T06:19:50.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/3818274235 wait complete. 2023-11-24T06:19:50.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.874+0000 7f3ee5ed5b80 20 client.5277 trim_cache size 0 max 16384 2023-11-24T06:19:50.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.892+0000 7f3ee5ed5b80 1 Processor -- start 2023-11-24T06:19:50.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.892+0000 7f3ee5ed5b80 1 -- start start 2023-11-24T06:19:50.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 0x5607d02b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0297080 0x5607d02d7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d02aced0 0x5607d02d7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d00698d0 con 0x5607d02aced0 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d02d3da0 con 0x5607d0297080 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607d02db690 con 0x5607d0294490 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d02aced0 0x5607d02d7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d02aced0 0x5607d02d7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37030/0 (socket says 172.21.15.89:37030) 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 -- 172.21.15.89:0/2694040739 learned_addr learned my addr 172.21.15.89:0/2694040739 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 -- 172.21.15.89:0/2694040739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 msgr2=0x5607d02b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/2694040739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 0x5607d02b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 -- 172.21.15.89:0/2694040739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0297080 msgr2=0x5607d02d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/2694040739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0297080 0x5607d02d7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 -- 172.21.15.89:0/2694040739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607d0293330 con 0x5607d02aced0 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.893+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/2694040739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d02aced0 0x5607d02d7dc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc003d90 tx=0x7f3ecc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3450ed228c5e7e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3eb9ffb700 1 -- 172.21.15.89:0/2694040739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00e040 con 0x5607d02aced0 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3eb9ffb700 1 -- 172.21.15.89:0/2694040739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc00f040 con 0x5607d02aced0 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3eb9ffb700 1 -- 172.21.15.89:0/2694040739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc021640 con 0x5607d02aced0 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2694040739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d02aced0 msgr2=0x5607d02d7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2694040739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d02aced0 0x5607d02d7dc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc003d90 tx=0x7f3ecc009750 comp rx=0 tx=0).stop 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2694040739 shutdown_connections 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2694040739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d02aced0 0x5607d02d7dc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2694040739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0294490 0x5607d02b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2694040739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d0297080 0x5607d02d7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2694040739 >> 172.21.15.89:0/2694040739 conn(0x5607d01c5310 msgr2=0x5607d02fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:50.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2694040739 shutdown_connections 2023-11-24T06:19:50.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.894+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2694040739 wait complete. 2023-11-24T06:19:50.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.895+0000 7f3ee5ed5b80 1 -- start start 2023-11-24T06:19:50.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.895+0000 7f3ee5ed5b80 1 Processor -- start 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0294490 0x5607d0344ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0297080 0x5607d032ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ee5ed5b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d0348eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d0348a90 con 0x5607d0294490 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d02dff90 con 0x5607d02aced0 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ee5ed5b80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607d03456f0 con 0x5607d0297080 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ee5ed5b80 10 client.0 _ll_register_callbacks cb 0x5607d00cfa30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0294490 0x5607d0344ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0294490 0x5607d0344ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37034/0 (socket says 172.21.15.89:37034) 2023-11-24T06:19:50.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 -- 172.21.15.89:0/2715623178 learned_addr learned my addr 172.21.15.89:0/2715623178 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0297080 0x5607d032ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d0348eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 -- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0297080 msgr2=0x5607d032ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0297080 0x5607d032ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 -- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 msgr2=0x5607d0348eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d0348eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 -- 172.21.15.89:0/2715623178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ecc003ac0 con 0x5607d0294490 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed3fff700 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d0348eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ed37fe700 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0294490 0x5607d0344ff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40017a0 tx=0x7f3ed40cff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579a6d4a5892177e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3eb27fc700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3eb27fc700 1 -- 172.21.15.89:0/2715623178 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4015cc0 con 0x5607d0294490 2023-11-24T06:19:50.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3eb27fc700 1 -- 172.21.15.89:0/2715623178 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec401d050 con 0x5607d0294490 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2715623178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ec4015940 con 0x5607d0294490 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.896+0000 7f3eb27fc700 1 -- 172.21.15.89:0/2715623178 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ec4015e20 con 0x5607d0294490 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ee5ed5b80 10 client.5283 fetch_fsmap learned FSMap version 17 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2715623178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5607cfffcc10 con 0x5607d0294490 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3eb27fc700 1 -- 172.21.15.89:0/2715623178 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec4015940 con 0x5607d0294490 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3eb27fc700 1 -- 172.21.15.89:0/2715623178 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3ec402f420 con 0x5607d0294490 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ee5ed5b80 10 client.5283 fetch_fsmap finished waiting for FSMap version 17 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ee5ed5b80 10 client.5283 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ee5ed5b80 20 client.5283 populate_metadata read hostname 'smithi089' 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] conn(0x5607d02d76a0 0x5607d02d7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:50.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ee5ed5b80 4 client.5283 mds_command: new command op to 5103 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2715623178 --> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5607d01b0f80 con 0x5607d02d76a0 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] conn(0x5607d02d76a0 0x5607d02d7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3eb27fc700 10 client.5283 ms_handle_connect on v2:172.21.15.123:6833/1601724762 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.897+0000 7f3ed2ffd700 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] conn(0x5607d02d76a0 0x5607d02d7a80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b67635053b4fec71 server_cookie=5ca84a7a66c39c8d in_seq=0 out_seq=0 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3eb27fc700 1 -- 172.21.15.89:0/2715623178 <== mds.0 v2:172.21.15.123:6833/1601724762 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5607d01b0f80 con 0x5607d02d76a0 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3eb27fc700 10 client.5283 handle_command_reply: tid=0 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 client.5283 shutdown 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2715623178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0294490 msgr2=0x5607d0344ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0294490 0x5607d0344ff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40017a0 tx=0x7f3ed40cff10 comp rx=0 tx=0).stop 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2715623178 shutdown_connections 2023-11-24T06:19:50.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:6833/1601724762,v1:172.21.15.123:6835/1601724762] conn(0x5607d02d76a0 0x5607d02d7a80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607d02aced0 0x5607d0348eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3eb27fc700 0 client.5283 ms_handle_reset on v2:172.21.15.123:6833/1601724762 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607d0294490 0x5607d0344ff0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 --2- 172.21.15.89:0/2715623178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5607d0297080 0x5607d032ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2715623178 >> 172.21.15.89:0/2715623178 conn(0x5607d01c5310 msgr2=0x5607d02daf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2715623178 shutdown_connections 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 1 -- 172.21.15.89:0/2715623178 wait complete. 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ee5ed5b80 20 client.5283 trim_cache size 0 max 16384 2023-11-24T06:19:50.908 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "be2488f4-68a7-42cc-93d1-ae9c6921fb48", 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 5103, 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 5.9163618080000004, 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 17, 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 34, 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 34, 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 14.279769396000001 2023-11-24T06:19:50.909 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:19:50.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3ec006ae00 msgr2=0x7f3ec006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3ec006ae00 0x7f3ec006d2c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4007e10 tx=0x7f3ec4009040 comp rx=0 tx=0).stop 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ed415e140 msgr2=0x7f3ed40cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ed415e140 0x7f3ed40cc030 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3ed406d1d0 tx=0x7f3ec8001120 comp rx=0 tx=0).stop 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/1458019424 shutdown_connections 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3ec006ae00 0x7f3ec006d2c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed414b5d0 0x7f3ed40b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3ebc004850 0x7f3ed415caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 --2- 172.21.15.89:0/1458019424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ed415e140 0x7f3ed40cc030 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/1458019424 >> 172.21.15.89:0/1458019424 conn(0x7f3ed40b57b0 msgr2=0x7f3ed40d0230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.898+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/1458019424 shutdown_connections 2023-11-24T06:19:50.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:50.899+0000 7f3ed9cdc700 1 -- 172.21.15.89:0/1458019424 wait complete. 2023-11-24T06:19:50.914 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:19:51.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.033+0000 7f06391e2700 1 Processor -- start 2023-11-24T06:19:51.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.033+0000 7f06391e2700 1 -- start start 2023-11-24T06:19:51.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.033+0000 7f06391e2700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f063415bdd0 0x7f06341604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:51.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.033+0000 7f06391e2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06340b3370 0x7f06340b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:51.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.033+0000 7f06391e2700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340b2030 con 0x7f06340d1c70 2023-11-24T06:19:51.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.033+0000 7f06391e2700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06340b2f50 con 0x7f06340b3370 2023-11-24T06:19:51.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.033+0000 7f06391e2700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340b22d0 con 0x7f063415bdd0 2023-11-24T06:19:51.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.033+0000 7f063359e700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f06340d1c70 0x7f06340ce6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40546/0 (socket says 172.21.15.89:40546) 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f063359e700 1 -- 172.21.15.89:0/849135186 learned_addr learned my addr 172.21.15.89:0/849135186 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0632d9d700 1 --2- 172.21.15.89:0/849135186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f063415bdd0 0x7f06341604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f063259c700 1 --2- 172.21.15.89:0/849135186 >> v2:172.21.15.123:3300/0 conn(0x7f06340b3370 0x7f06340b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0631d9b700 1 -- 172.21.15.89:0/849135186 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497156339 0 0) 0x7f06340b2030 con 0x7f06340d1c70 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0631d9b700 1 -- 172.21.15.89:0/849135186 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061c0035b0 con 0x7f06340d1c70 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0632d9d700 1 -- 172.21.15.89:0/849135186 >> v2:172.21.15.123:3300/0 conn(0x7f06340b3370 msgr2=0x7f06340b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0632d9d700 1 --2- 172.21.15.89:0/849135186 >> v2:172.21.15.123:3300/0 conn(0x7f06340b3370 0x7f06340b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0632d9d700 1 -- 172.21.15.89:0/849135186 >> v1:172.21.15.89:6789/0 conn(0x7f06340d1c70 legacy=0x7f06340ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0632d9d700 1 -- 172.21.15.89:0/849135186 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340b25b0 con 0x7f063415bdd0 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f063259c700 1 --2- 172.21.15.89:0/849135186 >> v2:172.21.15.123:3300/0 conn(0x7f06340b3370 0x7f06340b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0632d9d700 1 --2- 172.21.15.89:0/849135186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f063415bdd0 0x7f06341604e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0628009840 tx=0x7f0628009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a231b88653d1b02 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0631d9b700 1 -- 172.21.15.89:0/849135186 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628011830 con 0x7f063415bdd0 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0631d9b700 1 -- 172.21.15.89:0/849135186 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0628011990 con 0x7f063415bdd0 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.034+0000 7f0631d9b700 1 -- 172.21.15.89:0/849135186 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628011d00 con 0x7f063415bdd0 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.035+0000 7f06391e2700 1 -- 172.21.15.89:0/849135186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f063415bdd0 msgr2=0x7f06341604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.035+0000 7f06391e2700 1 --2- 172.21.15.89:0/849135186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f063415bdd0 0x7f06341604e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0628009840 tx=0x7f0628009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.035+0000 7f06391e2700 1 -- 172.21.15.89:0/849135186 shutdown_connections 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.035+0000 7f06391e2700 1 --2- 172.21.15.89:0/849135186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f063415bdd0 0x7f06341604e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.035+0000 7f06391e2700 1 --2- 172.21.15.89:0/849135186 >> v2:172.21.15.123:3300/0 conn(0x7f06340b3370 0x7f06340b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.035+0000 7f06391e2700 1 -- 172.21.15.89:0/849135186 >> 172.21.15.89:0/849135186 conn(0x7f06340731a0 msgr2=0x7f063409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.035+0000 7f06391e2700 1 -- 172.21.15.89:0/849135186 shutdown_connections 2023-11-24T06:19:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.035+0000 7f06391e2700 1 -- 172.21.15.89:0/849135186 wait complete. 2023-11-24T06:19:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.036+0000 7f06391e2700 1 Processor -- start 2023-11-24T06:19:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.036+0000 7f06391e2700 1 -- start start 2023-11-24T06:19:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f06391e2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06340b3370 0x7f063406cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f06391e2700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06340d1c70 0x7f06340cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f06391e2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f063414ff70 0x7f06341f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f06391e2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340c68a0 con 0x7f063414ff70 2023-11-24T06:19:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f06391e2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340c6b40 con 0x7f06340b3370 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f06391e2700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340caca0 con 0x7f06340d1c70 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f063359e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f063414ff70 0x7f06341f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f063359e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f063414ff70 0x7f06341f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37052/0 (socket says 172.21.15.89:37052) 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f063359e700 1 -- 172.21.15.89:0/363255447 learned_addr learned my addr 172.21.15.89:0/363255447 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f0632d9d700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06340b3370 0x7f063406cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f063259c700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06340d1c70 0x7f06340cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f063359e700 1 -- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06340d1c70 msgr2=0x7f06340cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f063359e700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06340d1c70 0x7f06340cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f063359e700 1 -- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06340b3370 msgr2=0x7f063406cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.037+0000 7f063359e700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06340b3370 0x7f063406cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f063359e700 1 -- 172.21.15.89:0/363255447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240094a0 con 0x7f063414ff70 2023-11-24T06:19:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f0632d9d700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06340b3370 0x7f063406cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f063259c700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06340d1c70 0x7f06340cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f063359e700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f063414ff70 0x7f06341f3d00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f062c00caa0 tx=0x7f062c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f061b7fe700 1 -- 172.21.15.89:0/363255447 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c00aa80 con 0x7f063414ff70 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f061b7fe700 1 -- 172.21.15.89:0/363255447 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c00abe0 con 0x7f063414ff70 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06280094a0 con 0x7f063414ff70 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f061b7fe700 1 -- 172.21.15.89:0/363255447 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c012400 con 0x7f063414ff70 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.038+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06340cb260 con 0x7f063414ff70 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.039+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0634067f60 con 0x7f063414ff70 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.039+0000 7f061b7fe700 1 -- 172.21.15.89:0/363255447 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f062c01b3b0 con 0x7f063414ff70 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.040+0000 7f061b7fe700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f062006aed0 0x7f062006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.040+0000 7f061b7fe700 1 -- 172.21.15.89:0/363255447 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f062c0892a0 con 0x7f063414ff70 2023-11-24T06:19:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.040+0000 7f0632d9d700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f062006aed0 0x7f062006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.040+0000 7f0632d9d700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f062006aed0 0x7f062006d390 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f06340b48a0 tx=0x7f0628008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:51.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.044+0000 7f061b7fe700 1 -- 172.21.15.89:0/363255447 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f062c0543e0 con 0x7f063414ff70 2023-11-24T06:19:51.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.278+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f06340b7c60 con 0x7f063414ff70 2023-11-24T06:19:51.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.279+0000 7f061b7fe700 1 -- 172.21.15.89:0/363255447 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f062c057a00 con 0x7f063414ff70 2023-11-24T06:19:51.282 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f062006aed0 msgr2=0x7f062006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f062006aed0 0x7f062006d390 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f06340b48a0 tx=0x7f0628008030 comp rx=0 tx=0).stop 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f063414ff70 msgr2=0x7f06341f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f063414ff70 0x7f06341f3d00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f062c00caa0 tx=0x7f062c00ce60 comp rx=0 tx=0).stop 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 shutdown_connections 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f063414ff70 0x7f06341f3d00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f062006aed0 0x7f062006d390 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06340b3370 0x7f063406cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 --2- 172.21.15.89:0/363255447 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06340d1c70 0x7f06340cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:51.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 >> 172.21.15.89:0/363255447 conn(0x7f06340731a0 msgr2=0x7f06340b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:51.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 shutdown_connections 2023-11-24T06:19:51.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:51.281+0000 7f06391e2700 1 -- 172.21.15.89:0/363255447 wait complete. 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:19:51.296 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:19:51.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.297 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:51.314 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42369sec preferred_lft 42369sec 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4376:7845:d8a4:7bc1/64 scope link noprefixroute 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 12:de:17:7b:11:37 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:19:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::3cfa:7aff:fed2:90c8/64 scope link 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 12:de:17:7b:11:37 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::10de:17ff:fe7b:1137/64 scope link 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 76:9a:20:1a:a1:88 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::749a:20ff:fe1a:a188/64 scope link 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:51.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.317 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:19:51.378 INFO:teuthology.orchestra.run.smithi089.stdout:1 2023-11-24T06:19:51.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.380 DEBUG:teuthology.orchestra.run.smithi089:> route 2023-11-24T06:19:51.438 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2023-11-24T06:19:51.438 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:19:51.438 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:19:51.438 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:19:51.439 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:19:51.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.439 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:19:51.439 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:19:51.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:19:51.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:19:51.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:19:51.439 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:19:51.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.544 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:19:51.560 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T06:19:51.561 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T06:19:51.561 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:19:51.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.561 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2023-11-24T06:19:51.627 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:51.627 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:51.684 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:19:51.685 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:51.685 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:51.741 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:19:51.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:19:51.827 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:19:51.827 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:19:51.827 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:51.836 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:19:51.836 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:19:51.836 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:51.836 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:51.837 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:51.912 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:19:51.912 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:19:51.912 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:19:51.912 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:51.912 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:51.913 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:19:51.913 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:19:51.913 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:19:51.914 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:19:51.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.914 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:19:51.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:51.943 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:52.032 INFO:teuthology.orchestra.run.smithi089.stdout:client4784 2023-11-24T06:19:52.032 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:19:52.032 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:52.051 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:19:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.209+0000 7f3e400f0700 1 Processor -- start 2023-11-24T06:19:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.209+0000 7f3e400f0700 1 -- start start 2023-11-24T06:19:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e400f0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e3815eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e400f0700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e3806c000 0x7f3e3806c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e400f0700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3806c990 con 0x7f3e38144660 2023-11-24T06:19:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e400f0700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380b4230 con 0x7f3e3806c000 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e400f0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380b3c40 con 0x7f3e3815ea60 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3e68d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e3806c000 0x7f3e3806c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3de8c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3e38144660 0x7f3e380b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40564/0 (socket says 172.21.15.89:40564) 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3de8c700 1 -- 172.21.15.89:0/3394642775 learned_addr learned my addr 172.21.15.89:0/3394642775 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3d68b700 1 --2- 172.21.15.89:0/3394642775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e3815eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3e68d700 1 -- 172.21.15.89:0/3394642775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 msgr2=0x7f3e3815eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3e68d700 1 --2- 172.21.15.89:0/3394642775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e3815eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3e68d700 1 -- 172.21.15.89:0/3394642775 >> v1:172.21.15.89:6789/0 conn(0x7f3e38144660 legacy=0x7f3e380b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3e68d700 1 -- 172.21.15.89:0/3394642775 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3815f520 con 0x7f3e3806c000 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3ce8a700 1 -- 172.21.15.89:0/3394642775 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460907676 0 0) 0x7f3e34001110 con 0x7f3e38144660 2023-11-24T06:19:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3e68d700 1 --2- 172.21.15.89:0/3394642775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3806c000 0x7f3e3806c450 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009d90 tx=0x7f3e3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d040e91f6dcca1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.210+0000 7f3e3ce8a700 1 -- 172.21.15.89:0/3394642775 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3000e070 con 0x7f3e3806c000 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e3ce8a700 1 -- 172.21.15.89:0/3394642775 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3000bbd0 con 0x7f3e3806c000 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e3ce8a700 1 -- 172.21.15.89:0/3394642775 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30012430 con 0x7f3e3806c000 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e400f0700 1 -- 172.21.15.89:0/3394642775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3806c000 msgr2=0x7f3e3806c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e400f0700 1 --2- 172.21.15.89:0/3394642775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3806c000 0x7f3e3806c450 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009d90 tx=0x7f3e3000b330 comp rx=0 tx=0).stop 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e400f0700 1 -- 172.21.15.89:0/3394642775 shutdown_connections 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e400f0700 1 --2- 172.21.15.89:0/3394642775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3806c000 0x7f3e3806c450 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e400f0700 1 --2- 172.21.15.89:0/3394642775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e3815eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e400f0700 1 -- 172.21.15.89:0/3394642775 >> 172.21.15.89:0/3394642775 conn(0x7f3e38071ac0 msgr2=0x7f3e38071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e400f0700 1 -- 172.21.15.89:0/3394642775 shutdown_connections 2023-11-24T06:19:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.211+0000 7f3e400f0700 1 -- 172.21.15.89:0/3394642775 wait complete. 2023-11-24T06:19:52.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.212+0000 7f3e400f0700 1 Processor -- start 2023-11-24T06:19:52.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.212+0000 7f3e400f0700 1 -- start start 2023-11-24T06:19:52.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e400f0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e3806c000 0x7f3e380add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:52.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e400f0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38144660 0x7f3e3814d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:52.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e400f0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e380ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e400f0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3815df20 con 0x7f3e38144660 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e400f0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38141aa0 con 0x7f3e3806c000 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e400f0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3813ea90 con 0x7f3e3815ea60 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3d68b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38144660 0x7f3e3814d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3d68b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38144660 0x7f3e3814d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37066/0 (socket says 172.21.15.89:37066) 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3d68b700 1 -- 172.21.15.89:0/969974200 learned_addr learned my addr 172.21.15.89:0/969974200 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3d68b700 1 -- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 msgr2=0x7f3e380ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3e68d700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e380ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3de8c700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e3806c000 0x7f3e380add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3d68b700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e380ae660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3d68b700 1 -- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e3806c000 msgr2=0x7f3e380add10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3d68b700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e3806c000 0x7f3e380add10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3d68b700 1 -- 172.21.15.89:0/969974200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e30009a70 con 0x7f3e38144660 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3e68d700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e380ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.213+0000 7f3e3de8c700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e3806c000 0x7f3e380add10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.214+0000 7f3e3d68b700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38144660 0x7f3e3814d550 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800e810 tx=0x7f3e2800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.214+0000 7f3e2e7fc700 1 -- 172.21.15.89:0/969974200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28013610 con 0x7f3e38144660 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.214+0000 7f3e2e7fc700 1 -- 172.21.15.89:0/969974200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28013770 con 0x7f3e38144660 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.214+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3813ed70 con 0x7f3e38144660 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.214+0000 7f3e2e7fc700 1 -- 172.21.15.89:0/969974200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28013b20 con 0x7f3e38144660 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.214+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e381417c0 con 0x7f3e38144660 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.215+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e380ae4b0 con 0x7f3e38144660 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.215+0000 7f3e2e7fc700 1 -- 172.21.15.89:0/969974200 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3e28013c80 con 0x7f3e38144660 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.215+0000 7f3e2e7fc700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.216+0000 7f3e2e7fc700 1 -- 172.21.15.89:0/969974200 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e28089310 con 0x7f3e38144660 2023-11-24T06:19:52.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.216+0000 7f3e3de8c700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:52.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.216+0000 7f3e3de8c700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e34007fd0 tx=0x7f3e34009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:52.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.220+0000 7f3e2e7fc700 1 -- 172.21.15.89:0/969974200 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e280543d0 con 0x7f3e38144660 2023-11-24T06:19:52.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.455+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3e380ae2b0 con 0x7f3e38144660 2023-11-24T06:19:52.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.456+0000 7f3e2e7fc700 1 -- 172.21.15.89:0/969974200 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3e280579f0 con 0x7f3e38144660 2023-11-24T06:19:52.459 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:19:52.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 msgr2=0x7f3e1c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:52.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e34007fd0 tx=0x7f3e34009040 comp rx=0 tx=0).stop 2023-11-24T06:19:52.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38144660 msgr2=0x7f3e3814d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:52.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38144660 0x7f3e3814d550 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800e810 tx=0x7f3e2800ebd0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 shutdown_connections 2023-11-24T06:19:52.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e3815ea60 0x7f3e380ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e3806c000 0x7f3e380add10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 --2- 172.21.15.89:0/969974200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38144660 0x7f3e3814d550 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:52.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 >> 172.21.15.89:0/969974200 conn(0x7f3e38071ac0 msgr2=0x7f3e3806db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:52.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 shutdown_connections 2023-11-24T06:19:52.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:52.458+0000 7f3e400f0700 1 -- 172.21.15.89:0/969974200 wait complete. 2023-11-24T06:19:52.472 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T06:19:52.473 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:19:52.473 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2023-11-24T06:19:52.473 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T06:19:52.473 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:52.473 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T06:19:52.473 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T06:19:52.473 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:19:52.473 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:19:52.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:52.473 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2023-11-24T06:19:52.495 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:19:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:19:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:19:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2023-11-24T06:19:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:52.495 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42359sec preferred_lft 42359sec 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::a981:dd7b:bd0d:851a/64 scope link noprefixroute 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 1a:69:ea:ee:dc:d5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:19:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::682d:1bff:fea0:f45c/64 scope link 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether d2:06:87:3a:6b:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::d006:87ff:fe3a:6bfb/64 scope link 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 1a:69:ea:ee:dc:d5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::1869:eaff:feee:dcd5/64 scope link 2023-11-24T06:19:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:19:52.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:52.498 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:19:52.566 INFO:teuthology.orchestra.run.smithi123.stdout:1 2023-11-24T06:19:52.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:52.568 DEBUG:teuthology.orchestra.run.smithi123:> route 2023-11-24T06:19:52.631 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2023-11-24T06:19:52.631 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:19:52.631 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:19:52.631 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:19:52.631 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:19:52.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:52.631 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:19:52.632 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:19:52.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:19:52.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:19:52.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:19:52.632 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:19:52.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:52.741 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:19:52.758 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T06:19:52.758 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T06:19:52.759 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:19:52.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:52.759 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2023-11-24T06:19:52.826 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:52.827 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:52.884 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:19:52.884 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:19:52.885 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:52.941 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:19:52.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:52.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:19:53.028 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:19:53.028 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:19:53.028 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:53.037 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:19:53.037 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:19:53.037 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:53.037 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:53.037 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:53.061 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:19:53.062 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:19:53.062 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:19:53.062 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:19:53.062 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:19:53.062 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:19:53.063 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:19:53.063 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:19:53.079 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:19:53.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:53.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:19:53.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:19:53.106 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:19:53.179 INFO:teuthology.orchestra.run.smithi123.stdout:client5023 2023-11-24T06:19:53.179 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:19:53.179 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:19:53.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T06:19:53.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.364+0000 7fb91b782700 1 Processor -- start 2023-11-24T06:19:53.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.364+0000 7fb91b782700 1 -- start start 2023-11-24T06:19:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb91b782700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9140d07c0 0x7fb9140d1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb91b782700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140d2530 con 0x7fb91415bd70 2023-11-24T06:19:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb91b782700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140aec00 con 0x7fb9140d20e0 2023-11-24T06:19:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb91b782700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140703c0 con 0x7fb9140d07c0 2023-11-24T06:19:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb91951e700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb91415bd70 0x7fb91406fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40578/0 (socket says 172.21.15.89:40578) 2023-11-24T06:19:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb91951e700 1 -- 172.21.15.89:0/4229383145 learned_addr learned my addr 172.21.15.89:0/4229383145 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901540176 0 0) 0x7fb9140d2530 con 0x7fb91415bd70 2023-11-24T06:19:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb918d1d700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9140d07c0 0x7fb9140d1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9000035b0 con 0x7fb91415bd70 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2011597082 0 0) 0x7fb9000035b0 con 0x7fb91415bd70 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9140d07c0 msgr2=0x7fb9140d1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9140d07c0 0x7fb9140d1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 >> v1:172.21.15.123:6789/0 conn(0x7fb9140d20e0 legacy=0x7fb9140af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140705d0 con 0x7fb91415bd70 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fb910001670 con 0x7fb91415bd70 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 >> v1:172.21.15.89:6789/0 conn(0x7fb91415bd70 legacy=0x7fb91406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb900003a30 0x7fb900003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb900004400 0x7fb9000048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.365+0000 7fb90bfff700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9140d07c0 0x7fb900005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140703c0 con 0x7fb900003a30 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90000ba40 con 0x7fb900004400 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90000bc00 con 0x7fb9140d07c0 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb91951e700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb900003a30 0x7fb900003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb918d1d700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb900004400 0x7fb9000048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb919d1f700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9140d07c0 0x7fb900005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb91951e700 1 -- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9140d07c0 msgr2=0x7fb900005130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb91951e700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9140d07c0 0x7fb900005130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb91951e700 1 -- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb900004400 msgr2=0x7fb9000048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb91951e700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb900004400 0x7fb9000048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb91951e700 1 -- 172.21.15.89:0/4229383145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9000035b0 con 0x7fb900003a30 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb91951e700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb900003a30 0x7fb900003e30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb91000c2f0 tx=0x7fb900005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a96da65a15e12e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91000ce30 con 0x7fb900003a30 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910005960 con 0x7fb900003a30 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.366+0000 7fb90bfff700 1 -- 172.21.15.89:0/4229383145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910005cf0 con 0x7fb900003a30 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.367+0000 7fb91b782700 1 -- 172.21.15.89:0/4229383145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb900003a30 msgr2=0x7fb900003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.367+0000 7fb91b782700 1 --2- 172.21.15.89:0/4229383145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb900003a30 0x7fb900003e30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb91000c2f0 tx=0x7fb900005880 comp rx=0 tx=0).stop 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.367+0000 7fb919d1f700 1 -- 172.21.15.89:0/4229383145 reap_dead start 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.367+0000 7fb91b782700 1 -- 172.21.15.89:0/4229383145 shutdown_connections 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.367+0000 7fb91b782700 1 -- 172.21.15.89:0/4229383145 >> 172.21.15.89:0/4229383145 conn(0x7fb91406dc80 msgr2=0x7fb91409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.367+0000 7fb91b782700 1 -- 172.21.15.89:0/4229383145 shutdown_connections 2023-11-24T06:19:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.367+0000 7fb91b782700 1 -- 172.21.15.89:0/4229383145 wait complete. 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.368+0000 7fb91b782700 1 Processor -- start 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.368+0000 7fb91b782700 1 -- start start 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb91b782700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb900003a30 0x7fb9141f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb91b782700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9140cb320 0x7fb9141f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb91b782700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9140cb7a0 0x7fb9141f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb91b782700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140c7380 con 0x7fb9140cb7a0 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb91b782700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9141f7f70 con 0x7fb9140cb320 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb91b782700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91406be20 con 0x7fb900003a30 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9140cb7a0 0x7fb9141f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9140cb7a0 0x7fb9141f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37090/0 (socket says 172.21.15.89:37090) 2023-11-24T06:19:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 -- 172.21.15.89:0/1352674398 learned_addr learned my addr 172.21.15.89:0/1352674398 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb91951e700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb900003a30 0x7fb9141f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb918d1d700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9140cb320 0x7fb9141f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 -- 172.21.15.89:0/1352674398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb900003a30 msgr2=0x7fb9141f4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb900003a30 0x7fb9141f4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 -- 172.21.15.89:0/1352674398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9140cb320 msgr2=0x7fb9141f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9140cb320 0x7fb9141f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 -- 172.21.15.89:0/1352674398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9141f7c90 con 0x7fb9140cb7a0 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb919d1f700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9140cb7a0 0x7fb9141f7580 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00d380 tx=0x7fb90c00d690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb909ffb700 1 -- 172.21.15.89:0/1352674398 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c00dc20 con 0x7fb9140cb7a0 2023-11-24T06:19:53.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb909ffb700 1 -- 172.21.15.89:0/1352674398 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c00dd80 con 0x7fb9140cb7a0 2023-11-24T06:19:53.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb909ffb700 1 -- 172.21.15.89:0/1352674398 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c003c40 con 0x7fb9140cb7a0 2023-11-24T06:19:53.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.369+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91406b7d0 con 0x7fb9140cb7a0 2023-11-24T06:19:53.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.370+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9141f6340 con 0x7fb9140cb7a0 2023-11-24T06:19:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.370+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91406b630 con 0x7fb9140cb7a0 2023-11-24T06:19:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.371+0000 7fb909ffb700 1 -- 172.21.15.89:0/1352674398 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb90c0050d0 con 0x7fb9140cb7a0 2023-11-24T06:19:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.371+0000 7fb909ffb700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8f806aed0 0x7fb8f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.371+0000 7fb909ffb700 1 -- 172.21.15.89:0/1352674398 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb90c089850 con 0x7fb9140cb7a0 2023-11-24T06:19:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.371+0000 7fb91951e700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8f806aed0 0x7fb8f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.372+0000 7fb91951e700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8f806aed0 0x7fb8f806d390 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb910001930 tx=0x7fb910007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.376+0000 7fb909ffb700 1 -- 172.21.15.89:0/1352674398 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb91406b630 con 0x7fb9140cb7a0 2023-11-24T06:19:53.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:53.608+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb91406b630 con 0x7fb9140cb7a0 2023-11-24T06:19:54.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.015+0000 7fb909ffb700 1 -- 172.21.15.89:0/1352674398 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb91406b630 con 0x7fb9140cb7a0 2023-11-24T06:19:54.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8f806aed0 msgr2=0x7fb8f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8f806aed0 0x7fb8f806d390 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb910001930 tx=0x7fb910007000 comp rx=0 tx=0).stop 2023-11-24T06:19:54.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9140cb7a0 msgr2=0x7fb9141f7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9140cb7a0 0x7fb9141f7580 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00d380 tx=0x7fb90c00d690 comp rx=0 tx=0).stop 2023-11-24T06:19:54.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 shutdown_connections 2023-11-24T06:19:54.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9140cb7a0 0x7fb9141f7580 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb900003a30 0x7fb9141f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8f806aed0 0x7fb8f806d390 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 --2- 172.21.15.89:0/1352674398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9140cb320 0x7fb9141f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 >> 172.21.15.89:0/1352674398 conn(0x7fb91406dc80 msgr2=0x7fb9140b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:54.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 shutdown_connections 2023-11-24T06:19:54.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.017+0000 7fb91b782700 1 -- 172.21.15.89:0/1352674398 wait complete. 2023-11-24T06:19:54.033 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T06:19:54.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.150+0000 7f62747ac700 1 Processor -- start 2023-11-24T06:19:54.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.151+0000 7f62747ac700 1 -- start start 2023-11-24T06:19:54.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.151+0000 7f62747ac700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 0x7f626c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.151+0000 7f62747ac700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f626c0af4d0 0x7f626c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.151+0000 7f62747ac700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c0ae430 con 0x7f626c15e080 2023-11-24T06:19:54.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.151+0000 7f62747ac700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c0af0b0 con 0x7f626c0af4d0 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.151+0000 7f62747ac700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0b05a0 con 0x7f626c06c710 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6272548700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f626c15e080 0x7f626c06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40598/0 (socket says 172.21.15.89:40598) 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6272548700 1 -- 172.21.15.89:0/733814324 learned_addr learned my addr 172.21.15.89:0/733814324 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271d47700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 0x7f626c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6272d49700 1 --2- 172.21.15.89:0/733814324 >> v2:172.21.15.123:3300/0 conn(0x7f626c0af4d0 0x7f626c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140250957 0 0) 0x7f626c0ae430 con 0x7f626c15e080 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62580035b0 con 0x7f626c15e080 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2402915654 0 0) 0x7f62580035b0 con 0x7f626c15e080 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 msgr2=0x7f626c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 0x7f626c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 >> v2:172.21.15.123:3300/0 conn(0x7f626c0af4d0 msgr2=0x7f626c0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 --2- 172.21.15.89:0/733814324 >> v2:172.21.15.123:3300/0 conn(0x7f626c0af4d0 0x7f626c0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c0ae710 con 0x7f626c15e080 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6272d49700 1 --2- 172.21.15.89:0/733814324 >> v2:172.21.15.123:3300/0 conn(0x7f626c0af4d0 0x7f626c0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f625c001670 con 0x7f626c15e080 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 >> v1:172.21.15.89:6789/0 conn(0x7f626c15e080 legacy=0x7f626c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258003a30 0x7f6258003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 0x7f6258004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6258004d90 0x7f6258005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0af0b0 con 0x7f6258004d90 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0b05a0 con 0x7f6258003a30 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625800ba10 con 0x7f626c06c710 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6271d47700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 0x7f6258004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6272548700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258003a30 0x7f6258003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.152+0000 7f6272d49700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6258004d90 0x7f6258005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6272d49700 1 -- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 msgr2=0x7f6258004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:54.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6272d49700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 0x7f6258004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6272d49700 1 -- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258003a30 msgr2=0x7f6258003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:54.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6272d49700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258003a30 0x7f6258003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6272d49700 1 -- 172.21.15.89:0/733814324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580035b0 con 0x7f6258004d90 2023-11-24T06:19:54.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6271d47700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c06c710 0x7f6258004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:19:54.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6272d49700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6258004d90 0x7f6258005210 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f626400e380 tx=0x7f6258005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1209f232cdd06fe6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626400ee20 con 0x7f6258004d90 2023-11-24T06:19:54.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626400f040 con 0x7f6258004d90 2023-11-24T06:19:54.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6271546700 1 -- 172.21.15.89:0/733814324 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62640146a0 con 0x7f6258004d90 2023-11-24T06:19:54.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f62747ac700 1 -- 172.21.15.89:0/733814324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6258004d90 msgr2=0x7f6258005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f62747ac700 1 --2- 172.21.15.89:0/733814324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6258004d90 0x7f6258005210 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f626400e380 tx=0x7f6258005880 comp rx=0 tx=0).stop 2023-11-24T06:19:54.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.153+0000 7f6272d49700 1 -- 172.21.15.89:0/733814324 reap_dead start 2023-11-24T06:19:54.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.154+0000 7f62747ac700 1 -- 172.21.15.89:0/733814324 shutdown_connections 2023-11-24T06:19:54.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.154+0000 7f62747ac700 1 -- 172.21.15.89:0/733814324 >> 172.21.15.89:0/733814324 conn(0x7f626c071b90 msgr2=0x7f626c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:54.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.154+0000 7f62747ac700 1 -- 172.21.15.89:0/733814324 shutdown_connections 2023-11-24T06:19:54.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.154+0000 7f62747ac700 1 -- 172.21.15.89:0/733814324 wait complete. 2023-11-24T06:19:54.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.155+0000 7f62747ac700 1 Processor -- start 2023-11-24T06:19:54.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.155+0000 7f62747ac700 1 -- start start 2023-11-24T06:19:54.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.155+0000 7f62747ac700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258004d90 0x7f626c15f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f62747ac700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f626c156ca0 0x7f626c14e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f62747ac700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c151700 0x7f626c151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f62747ac700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c1501e0 con 0x7f626c156ca0 2023-11-24T06:19:54.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f62747ac700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c1520d0 con 0x7f6258004d90 2023-11-24T06:19:54.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f62747ac700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c14f940 con 0x7f626c151700 2023-11-24T06:19:54.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f626c156ca0 0x7f626c14e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f626c156ca0 0x7f626c14e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37108/0 (socket says 172.21.15.89:37108) 2023-11-24T06:19:54.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 -- 172.21.15.89:0/967515557 learned_addr learned my addr 172.21.15.89:0/967515557 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:54.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6272548700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258004d90 0x7f626c15f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6272d49700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c151700 0x7f626c151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 -- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c151700 msgr2=0x7f626c151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c151700 0x7f626c151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 -- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258004d90 msgr2=0x7f626c15f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258004d90 0x7f626c15f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 -- 172.21.15.89:0/967515557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c14fc20 con 0x7f626c156ca0 2023-11-24T06:19:54.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6272d49700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c151700 0x7f626c151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:54.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6271d47700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f626c156ca0 0x7f626c14e6f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6268011ca0 tx=0x7f626800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6262ffd700 1 -- 172.21.15.89:0/967515557 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268011e40 con 0x7f626c156ca0 2023-11-24T06:19:54.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6262ffd700 1 -- 172.21.15.89:0/967515557 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626800fd40 con 0x7f626c156ca0 2023-11-24T06:19:54.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c14ff00 con 0x7f626c156ca0 2023-11-24T06:19:54.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f6262ffd700 1 -- 172.21.15.89:0/967515557 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268004260 con 0x7f626c156ca0 2023-11-24T06:19:54.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.156+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c14ec30 con 0x7f626c156ca0 2023-11-24T06:19:54.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.157+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c0753f0 con 0x7f626c156ca0 2023-11-24T06:19:54.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.158+0000 7f6262ffd700 1 -- 172.21.15.89:0/967515557 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f62680043c0 con 0x7f626c156ca0 2023-11-24T06:19:54.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.158+0000 7f6262ffd700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f625006aed0 0x7f625006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.158+0000 7f6262ffd700 1 -- 172.21.15.89:0/967515557 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6268089cb0 con 0x7f626c156ca0 2023-11-24T06:19:54.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.158+0000 7f6272548700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f625006aed0 0x7f625006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.158+0000 7f6272548700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f625006aed0 0x7f625006d390 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f625c000c00 tx=0x7f625c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.163+0000 7f6262ffd700 1 -- 172.21.15.89:0/967515557 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6268054d60 con 0x7f626c156ca0 2023-11-24T06:19:54.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.384+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f626c15f910 con 0x7f625006aed0 2023-11-24T06:19:54.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.385+0000 7f6262ffd700 1 -- 172.21.15.89:0/967515557 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f626c15f910 con 0x7f625006aed0 2023-11-24T06:19:54.388 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-11-24T06:19:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2023-11-24T06:19:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:54.389 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-11-24T06:19:54.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.387+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f625006aed0 msgr2=0x7f625006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.387+0000 7f62747ac700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f625006aed0 0x7f625006d390 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f625c000c00 tx=0x7f625c009040 comp rx=0 tx=0).stop 2023-11-24T06:19:54.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f626c156ca0 msgr2=0x7f626c14e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f626c156ca0 0x7f626c14e6f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6268011ca0 tx=0x7f626800f520 comp rx=0 tx=0).stop 2023-11-24T06:19:54.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 shutdown_connections 2023-11-24T06:19:54.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f626c151700 0x7f626c151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f625006aed0 0x7f625006d390 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6258004d90 0x7f626c15f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 --2- 172.21.15.89:0/967515557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f626c156ca0 0x7f626c14e6f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 >> 172.21.15.89:0/967515557 conn(0x7f626c071b90 msgr2=0x7f626c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:54.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 shutdown_connections 2023-11-24T06:19:54.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.388+0000 7f62747ac700 1 -- 172.21.15.89:0/967515557 wait complete. 2023-11-24T06:19:54.402 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T06:19:54.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.522+0000 7fe3917d8700 1 Processor -- start 2023-11-24T06:19:54.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.522+0000 7fe3917d8700 1 -- start start 2023-11-24T06:19:54.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.523+0000 7fe3917d8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0746a0 0x7fe38c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.523+0000 7fe3917d8700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe38c0afc90 0x7fe38c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.523+0000 7fe3917d8700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c0b3c60 con 0x7fe38c15e160 2023-11-24T06:19:54.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.523+0000 7fe3917d8700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe38c0b4500 con 0x7fe38c0afc90 2023-11-24T06:19:54.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.523+0000 7fe3917d8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0b58c0 con 0x7fe38c0746a0 2023-11-24T06:19:54.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.523+0000 7fe38affd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe38c15e160 0x7fe38c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40618/0 (socket says 172.21.15.89:40618) 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.523+0000 7fe38affd700 1 -- 172.21.15.89:0/4151171162 learned_addr learned my addr 172.21.15.89:0/4151171162 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38b7fe700 1 --2- 172.21.15.89:0/4151171162 >> v2:172.21.15.123:3300/0 conn(0x7fe38c0afc90 0x7fe38c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562725498 0 0) 0x7fe38c0b3c60 con 0x7fe38c15e160 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0746a0 0x7fe38c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3740035b0 con 0x7fe38c15e160 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3992945946 0 0) 0x7fe3740035b0 con 0x7fe38c15e160 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0746a0 msgr2=0x7fe38c0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0746a0 0x7fe38c0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 >> v2:172.21.15.123:3300/0 conn(0x7fe38c0afc90 msgr2=0x7fe38c06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:54.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 --2- 172.21.15.89:0/4151171162 >> v2:172.21.15.123:3300/0 conn(0x7fe38c0afc90 0x7fe38c06d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0b0140 con 0x7fe38c15e160 2023-11-24T06:19:54.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fe380001670 con 0x7fe38c15e160 2023-11-24T06:19:54.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 >> v1:172.21.15.89:6789/0 conn(0x7fe38c15e160 legacy=0x7fe38c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0746a0 0x7fe374003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe374004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe374004d90 0x7fe374005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0b4500 con 0x7fe374003f80 2023-11-24T06:19:54.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0b58c0 con 0x7fe374004d90 2023-11-24T06:19:54.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37400ba10 con 0x7fe38c0746a0 2023-11-24T06:19:54.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe374004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38affd700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0746a0 0x7fe374003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38b7fe700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe374004d90 0x7fe374005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38a7fc700 1 -- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0746a0 msgr2=0x7fe374003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:54.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0746a0 0x7fe374003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38a7fc700 1 -- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe374004d90 msgr2=0x7fe374005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:54.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe374004d90 0x7fe374005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38a7fc700 1 -- 172.21.15.89:0/4151171162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3740035b0 con 0x7fe374003f80 2023-11-24T06:19:54.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38b7fe700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe374004d90 0x7fe374005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:54.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.524+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe374004850 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009b60 tx=0x7fe374005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83596eeabbcb83b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c00e660 con 0x7fe374003f80 2023-11-24T06:19:54.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c00e7c0 con 0x7fe374003f80 2023-11-24T06:19:54.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe389ffb700 1 -- 172.21.15.89:0/4151171162 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c00eb30 con 0x7fe374003f80 2023-11-24T06:19:54.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe3917d8700 1 -- 172.21.15.89:0/4151171162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 msgr2=0x7fe374004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe3917d8700 1 --2- 172.21.15.89:0/4151171162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe374004850 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009b60 tx=0x7fe374005880 comp rx=0 tx=0).stop 2023-11-24T06:19:54.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe38b7fe700 1 -- 172.21.15.89:0/4151171162 reap_dead start 2023-11-24T06:19:54.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe3917d8700 1 -- 172.21.15.89:0/4151171162 shutdown_connections 2023-11-24T06:19:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe3917d8700 1 -- 172.21.15.89:0/4151171162 >> 172.21.15.89:0/4151171162 conn(0x7fe38c0d0bf0 msgr2=0x7fe38c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe3917d8700 1 -- 172.21.15.89:0/4151171162 shutdown_connections 2023-11-24T06:19:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.525+0000 7fe3917d8700 1 -- 172.21.15.89:0/4151171162 wait complete. 2023-11-24T06:19:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.526+0000 7fe3917d8700 1 Processor -- start 2023-11-24T06:19:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe3917d8700 1 -- start start 2023-11-24T06:19:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe3917d8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe38c0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe3917d8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0c6d10 0x7fe38c0c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe3917d8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c0c8cc0 0x7fe38c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe3917d8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0c88a0 con 0x7fe374003f80 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe3917d8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0c76d0 con 0x7fe38c0c8cc0 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe3917d8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0c7940 con 0x7fe38c0c6d10 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe38affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe38c0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe38affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe38c0ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37126/0 (socket says 172.21.15.89:37126) 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe38affd700 1 -- 172.21.15.89:0/2850396256 learned_addr learned my addr 172.21.15.89:0/2850396256 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0c6d10 0x7fe38c0c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.527+0000 7fe38b7fe700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c0c8cc0 0x7fe38c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe38affd700 1 -- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0c6d10 msgr2=0x7fe38c0c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe38affd700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0c6d10 0x7fe38c0c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe38affd700 1 -- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c0c8cc0 msgr2=0x7fe38c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe38affd700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c0c8cc0 0x7fe38c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe38affd700 1 -- 172.21.15.89:0/2850396256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0c7c20 con 0x7fe374003f80 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0c6d10 0x7fe38c0c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe38b7fe700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c0c8cc0 0x7fe38c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe38affd700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe38c0ca5d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe380007310 tx=0x7fe380007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe3737fe700 1 -- 172.21.15.89:0/2850396256 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380007800 con 0x7fe374003f80 2023-11-24T06:19:54.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe3737fe700 1 -- 172.21.15.89:0/2850396256 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38000f8d0 con 0x7fe374003f80 2023-11-24T06:19:54.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c0c7f00 con 0x7fe374003f80 2023-11-24T06:19:54.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe3737fe700 1 -- 172.21.15.89:0/2850396256 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38000fc60 con 0x7fe374003f80 2023-11-24T06:19:54.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.528+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c0c84c0 con 0x7fe374003f80 2023-11-24T06:19:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.529+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c068fb0 con 0x7fe374003f80 2023-11-24T06:19:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.529+0000 7fe3737fe700 1 -- 172.21.15.89:0/2850396256 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe38001d460 con 0x7fe374003f80 2023-11-24T06:19:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.530+0000 7fe3737fe700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe37806aed0 0x7fe37806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.530+0000 7fe3737fe700 1 -- 172.21.15.89:0/2850396256 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe380089600 con 0x7fe374003f80 2023-11-24T06:19:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.530+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe37806aed0 0x7fe37806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.530+0000 7fe38a7fc700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe37806aed0 0x7fe37806d390 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0090b0 tx=0x7fe37c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.534+0000 7fe3737fe700 1 -- 172.21.15.89:0/2850396256 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe380054740 con 0x7fe374003f80 2023-11-24T06:19:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.724+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe38c06b7e0 con 0x7fe374003f80 2023-11-24T06:19:54.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.725+0000 7fe3737fe700 1 -- 172.21.15.89:0/2850396256 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe380057d60 con 0x7fe374003f80 2023-11-24T06:19:54.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe37806aed0 msgr2=0x7fe37806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe37806aed0 0x7fe37806d390 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0090b0 tx=0x7fe37c008030 comp rx=0 tx=0).stop 2023-11-24T06:19:54.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 msgr2=0x7fe38c0ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe38c0ca5d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe380007310 tx=0x7fe380007620 comp rx=0 tx=0).stop 2023-11-24T06:19:54.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 shutdown_connections 2023-11-24T06:19:54.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe37806aed0 0x7fe37806d390 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c0c8cc0 0x7fe38c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe374003f80 0x7fe38c0ca5d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 --2- 172.21.15.89:0/2850396256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe38c0c6d10 0x7fe38c0c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 >> 172.21.15.89:0/2850396256 conn(0x7fe38c0d0bf0 msgr2=0x7fe38c15d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:54.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 shutdown_connections 2023-11-24T06:19:54.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.727+0000 7fe3917d8700 1 -- 172.21.15.89:0/2850396256 wait complete. 2023-11-24T06:19:54.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create vol3975 2023-11-24T06:19:54.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.861+0000 7f7eb2df1700 1 Processor -- start 2023-11-24T06:19:54.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.861+0000 7f7eb2df1700 1 -- start start 2023-11-24T06:19:54.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.862+0000 7f7eb2df1700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7eac15e160 0x7f7eac0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.862+0000 7f7eb2df1700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac0b3c60 con 0x7f7eac0746a0 2023-11-24T06:19:54.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.862+0000 7f7eb2df1700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac0b58c0 con 0x7f7eac0afc90 2023-11-24T06:19:54.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.862+0000 7f7eb2df1700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac0b4500 con 0x7f7eac15e160 2023-11-24T06:19:54.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.862+0000 7f7eb0b8d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7eac15e160 0x7f7eac0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.862+0000 7f7eb138e700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7eac0746a0 0x7f7eac06d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40640/0 (socket says 172.21.15.89:40640) 2023-11-24T06:19:54.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.862+0000 7f7eb138e700 1 -- 172.21.15.89:0/4169290476 learned_addr learned my addr 172.21.15.89:0/4169290476 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:54.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eb0b8d700 1 -- 172.21.15.89:0/4169290476 >> v1:172.21.15.123:6789/0 conn(0x7f7eac0afc90 legacy=0x7f7eac073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eb0b8d700 1 -- 172.21.15.89:0/4169290476 >> v1:172.21.15.89:6789/0 conn(0x7f7eac0746a0 legacy=0x7f7eac06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eb0b8d700 1 -- 172.21.15.89:0/4169290476 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac0b0140 con 0x7f7eac15e160 2023-11-24T06:19:54.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eab7fe700 1 -- 172.21.15.89:0/4169290476 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070368419 0 0) 0x7f7eac0b3c60 con 0x7f7eac0746a0 2023-11-24T06:19:54.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eb0b8d700 1 --2- 172.21.15.89:0/4169290476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac0b5da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009840 tx=0x7f7ea0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a256b2e9ca9fe7b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eab7fe700 1 -- 172.21.15.89:0/4169290476 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0011830 con 0x7f7eac15e160 2023-11-24T06:19:54.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eab7fe700 1 -- 172.21.15.89:0/4169290476 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea0011990 con 0x7f7eac15e160 2023-11-24T06:19:54.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eab7fe700 1 -- 172.21.15.89:0/4169290476 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0011d20 con 0x7f7eac15e160 2023-11-24T06:19:54.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eb2df1700 1 -- 172.21.15.89:0/4169290476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 msgr2=0x7f7eac0b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eb2df1700 1 --2- 172.21.15.89:0/4169290476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac0b5da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009840 tx=0x7f7ea0009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:54.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eb2df1700 1 -- 172.21.15.89:0/4169290476 shutdown_connections 2023-11-24T06:19:54.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.863+0000 7f7eb2df1700 1 --2- 172.21.15.89:0/4169290476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac0b5da0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.864+0000 7f7eb2df1700 1 -- 172.21.15.89:0/4169290476 >> 172.21.15.89:0/4169290476 conn(0x7f7eac0d0bf0 msgr2=0x7f7eac0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:54.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.864+0000 7f7eb2df1700 1 -- 172.21.15.89:0/4169290476 shutdown_connections 2023-11-24T06:19:54.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.864+0000 7f7eb2df1700 1 -- 172.21.15.89:0/4169290476 wait complete. 2023-11-24T06:19:54.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.865+0000 7f7eb2df1700 1 Processor -- start 2023-11-24T06:19:54.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.865+0000 7f7eb2df1700 1 -- start start 2023-11-24T06:19:54.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb2df1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eac0746a0 0x7f7eac157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb2df1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eac0afc90 0x7f7eac155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb2df1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac160a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb2df1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac157100 con 0x7f7eac15e160 2023-11-24T06:19:54.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb2df1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac155f90 con 0x7f7eac0746a0 2023-11-24T06:19:54.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb2df1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac156480 con 0x7f7eac0afc90 2023-11-24T06:19:54.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb138e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac160a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb138e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac160a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37144/0 (socket says 172.21.15.89:37144) 2023-11-24T06:19:54.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb138e700 1 -- 172.21.15.89:0/233181541 learned_addr learned my addr 172.21.15.89:0/233181541 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:54.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eb0b8d700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eac0746a0 0x7f7eac157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.866+0000 7f7eabfff700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eac0afc90 0x7f7eac155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eb138e700 1 -- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eac0afc90 msgr2=0x7f7eac155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eb138e700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eac0afc90 0x7f7eac155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eb138e700 1 -- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eac0746a0 msgr2=0x7f7eac157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:54.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eb138e700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eac0746a0 0x7f7eac157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:54.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eb138e700 1 -- 172.21.15.89:0/233181541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea00094a0 con 0x7f7eac15e160 2023-11-24T06:19:54.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eabfff700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eac0afc90 0x7f7eac155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:54.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eb138e700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac160a80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400cc10 tx=0x7f7ea400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7ea97fa700 1 -- 172.21.15.89:0/233181541 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea400ab10 con 0x7f7eac15e160 2023-11-24T06:19:54.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7ea97fa700 1 -- 172.21.15.89:0/233181541 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea400ac90 con 0x7f7eac15e160 2023-11-24T06:19:54.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac156760 con 0x7f7eac15e160 2023-11-24T06:19:54.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7ea97fa700 1 -- 172.21.15.89:0/233181541 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea40070d0 con 0x7f7eac15e160 2023-11-24T06:19:54.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.867+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac156d20 con 0x7f7eac15e160 2023-11-24T06:19:54.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.868+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eac157240 con 0x7f7eac15e160 2023-11-24T06:19:54.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.868+0000 7f7ea97fa700 1 -- 172.21.15.89:0/233181541 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7ea4007230 con 0x7f7eac15e160 2023-11-24T06:19:54.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.869+0000 7f7ea97fa700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9006aed0 0x7f7e9006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:54.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.869+0000 7f7ea97fa700 1 -- 172.21.15.89:0/233181541 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ea4089380 con 0x7f7eac15e160 2023-11-24T06:19:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.869+0000 7f7eb0b8d700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9006aed0 0x7f7e9006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.869+0000 7f7eb0b8d700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9006aed0 0x7f7e9006d390 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7ea00090b0 tx=0x7f7ea001c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:54.873+0000 7f7ea97fa700 1 -- 172.21.15.89:0/233181541 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7eac157240 con 0x7f7eac15e160 2023-11-24T06:19:55.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:55.091+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "vol3975", "target": ["mon-mgr", ""]}) v1 -- 0x7f7eac0b4190 con 0x7f7e9006aed0 2023-11-24T06:19:58.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.029+0000 7f7ea97fa700 1 -- 172.21.15.89:0/233181541 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 Volume created successfully (no MDS daemons created)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f7ea0006c80 con 0x7f7e9006aed0 2023-11-24T06:19:58.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.033+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9006aed0 msgr2=0x7f7e9006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.033+0000 7f7eb2df1700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9006aed0 0x7f7e9006d390 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7ea00090b0 tx=0x7f7ea001c040 comp rx=0 tx=0).stop 2023-11-24T06:19:58.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.033+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 msgr2=0x7f7eac160a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.033+0000 7f7eb2df1700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac160a80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400cc10 tx=0x7f7ea400cf20 comp rx=0 tx=0).stop 2023-11-24T06:19:58.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.034+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 shutdown_connections 2023-11-24T06:19:58.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.034+0000 7f7eb2df1700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac15e160 0x7f7eac160a80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.034+0000 7f7eb2df1700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7e9006aed0 0x7f7e9006d390 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.034+0000 7f7eb2df1700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eac0746a0 0x7f7eac157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.034+0000 7f7eb2df1700 1 --2- 172.21.15.89:0/233181541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7eac0afc90 0x7f7eac155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.034+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 >> 172.21.15.89:0/233181541 conn(0x7f7eac0d0bf0 msgr2=0x7f7eac15ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:58.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.034+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 shutdown_connections 2023-11-24T06:19:58.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.035+0000 7f7eb2df1700 1 -- 172.21.15.89:0/233181541 wait complete. 2023-11-24T06:19:58.040 INFO:teuthology.orchestra.run.smithi089.stderr:Volume created successfully (no MDS daemons created) 2023-11-24T06:19:58.060 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T06:19:58.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.185+0000 7fdf625d9700 1 Processor -- start 2023-11-24T06:19:58.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.185+0000 7fdf625d9700 1 -- start start 2023-11-24T06:19:58.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf625d9700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf5c15e2a0 0x7fdf5c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf625d9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c0ce090 0x7fdf5c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf625d9700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c06e7f0 con 0x7fdf5c073f90 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf625d9700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5c06dec0 con 0x7fdf5c15e2a0 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf625d9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c06e1a0 con 0x7fdf5c0ce090 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5bfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf5c15e2a0 0x7fdf5c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5bfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf5c15e2a0 0x7fdf5c06ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37162/0 (socket says 172.21.15.89:37162) 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5bfff700 1 -- 172.21.15.89:0/2834526606 learned_addr learned my addr 172.21.15.89:0/2834526606 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5b7fe700 1 --2- 172.21.15.89:0/2834526606 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c0ce090 0x7fdf5c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5bfff700 1 -- 172.21.15.89:0/2834526606 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c0ce090 msgr2=0x7fdf5c06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5bfff700 1 --2- 172.21.15.89:0/2834526606 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c0ce090 0x7fdf5c06ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5bfff700 1 -- 172.21.15.89:0/2834526606 >> v1:172.21.15.123:6789/0 conn(0x7fdf5c073f90 legacy=0x7fdf5c0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5bfff700 1 -- 172.21.15.89:0/2834526606 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c06e480 con 0x7fdf5c15e2a0 2023-11-24T06:19:58.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.186+0000 7fdf5bfff700 1 --2- 172.21.15.89:0/2834526606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c15e2a0 0x7fdf5c06ba70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdf500098f0 tx=0x7fdf50009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739eea66bde4359e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf5affd700 1 -- 172.21.15.89:0/2834526606 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50004230 con 0x7fdf5c15e2a0 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf5affd700 1 -- 172.21.15.89:0/2834526606 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50004390 con 0x7fdf5c15e2a0 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf5affd700 1 -- 172.21.15.89:0/2834526606 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf500036a0 con 0x7fdf5c15e2a0 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf625d9700 1 -- 172.21.15.89:0/2834526606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c15e2a0 msgr2=0x7fdf5c06ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf625d9700 1 --2- 172.21.15.89:0/2834526606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c15e2a0 0x7fdf5c06ba70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdf500098f0 tx=0x7fdf50009c00 comp rx=0 tx=0).stop 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf625d9700 1 -- 172.21.15.89:0/2834526606 shutdown_connections 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf625d9700 1 --2- 172.21.15.89:0/2834526606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c15e2a0 0x7fdf5c06ba70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf625d9700 1 --2- 172.21.15.89:0/2834526606 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c0ce090 0x7fdf5c06ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf625d9700 1 -- 172.21.15.89:0/2834526606 >> 172.21.15.89:0/2834526606 conn(0x7fdf5c0cd570 msgr2=0x7fdf5c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf625d9700 1 -- 172.21.15.89:0/2834526606 shutdown_connections 2023-11-24T06:19:58.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.187+0000 7fdf625d9700 1 -- 172.21.15.89:0/2834526606 wait complete. 2023-11-24T06:19:58.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.188+0000 7fdf625d9700 1 Processor -- start 2023-11-24T06:19:58.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.188+0000 7fdf625d9700 1 -- start start 2023-11-24T06:19:58.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf625d9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c073f90 0x7fdf5c15b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf625d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c0ce090 0x7fdf5c15c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf625d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf5c15e2a0 0x7fdf5c158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf625d9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c15b6f0 con 0x7fdf5c0ce090 2023-11-24T06:19:58.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf625d9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c15b9d0 con 0x7fdf5c15e2a0 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf625d9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c15bcb0 con 0x7fdf5c073f90 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c0ce090 0x7fdf5c15c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c0ce090 0x7fdf5c15c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37168/0 (socket says 172.21.15.89:37168) 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf5b7fe700 1 -- 172.21.15.89:0/708154507 learned_addr learned my addr 172.21.15.89:0/708154507 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf5bfff700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c073f90 0x7fdf5c15b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.189+0000 7fdf60b76700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf5c15e2a0 0x7fdf5c158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf5b7fe700 1 -- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c073f90 msgr2=0x7fdf5c15b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf5b7fe700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c073f90 0x7fdf5c15b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf5b7fe700 1 -- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf5c15e2a0 msgr2=0x7fdf5c158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf5b7fe700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf5c15e2a0 0x7fdf5c158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf5b7fe700 1 -- 172.21.15.89:0/708154507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf50009550 con 0x7fdf5c0ce090 2023-11-24T06:19:58.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf60b76700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf5c15e2a0 0x7fdf5c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf5bfff700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c073f90 0x7fdf5c15b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf5b7fe700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c0ce090 0x7fdf5c15c790 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00c7f0 tx=0x7fdf4c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf58ff9700 1 -- 172.21.15.89:0/708154507 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c011800 con 0x7fdf5c0ce090 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf58ff9700 1 -- 172.21.15.89:0/708154507 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c011960 con 0x7fdf5c0ce090 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c15bf90 con 0x7fdf5c0ce090 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf58ff9700 1 -- 172.21.15.89:0/708154507 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c011cf0 con 0x7fdf5c0ce090 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.190+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c159300 con 0x7fdf5c0ce090 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.191+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c067f60 con 0x7fdf5c0ce090 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.191+0000 7fdf58ff9700 1 -- 172.21.15.89:0/708154507 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdf4c018360 con 0x7fdf5c0ce090 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.192+0000 7fdf58ff9700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf4806aed0 0x7fdf4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.192+0000 7fdf58ff9700 1 -- 172.21.15.89:0/708154507 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6543+0+0 (secure 0 0 0) 0x7fdf4c089920 con 0x7fdf5c0ce090 2023-11-24T06:19:58.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.192+0000 7fdf5bfff700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf4806aed0 0x7fdf4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.192+0000 7fdf5bfff700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf4806aed0 0x7fdf4806d390 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009130 tx=0x7fdf50008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.196+0000 7fdf58ff9700 1 -- 172.21.15.89:0/708154507 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf4c0546a0 con 0x7fdf5c0ce090 2023-11-24T06:19:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.414+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf5c0ce650 con 0x7fdf4806aed0 2023-11-24T06:19:58.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.415+0000 7fdf58ff9700 1 -- 172.21.15.89:0/708154507 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+80 (secure 0 0 0) 0x7fdf50003850 con 0x7fdf4806aed0 2023-11-24T06:19:58.418 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-11-24T06:19:58.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:58.418 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2023-11-24T06:19:58.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:19:58.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:19:58.418 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "vol3975" 2023-11-24T06:19:58.418 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:19:58.419 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-11-24T06:19:58.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf4806aed0 msgr2=0x7fdf4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf4806aed0 0x7fdf4806d390 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009130 tx=0x7fdf50008030 comp rx=0 tx=0).stop 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c0ce090 msgr2=0x7fdf5c15c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c0ce090 0x7fdf5c15c790 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00c7f0 tx=0x7fdf4c00cbb0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 shutdown_connections 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdf4806aed0 0x7fdf4806d390 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf5c15e2a0 0x7fdf5c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf5c073f90 0x7fdf5c15b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 --2- 172.21.15.89:0/708154507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf5c0ce090 0x7fdf5c15c790 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.417+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 >> 172.21.15.89:0/708154507 conn(0x7fdf5c0cd570 msgr2=0x7fdf5c071bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.418+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 shutdown_connections 2023-11-24T06:19:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.418+0000 7fdf625d9700 1 -- 172.21.15.89:0/708154507 wait complete. 2023-11-24T06:19:58.433 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool ls detail --format=json 2023-11-24T06:19:58.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.552+0000 7f3d1256f700 1 Processor -- start 2023-11-24T06:19:58.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.552+0000 7f3d1256f700 1 -- start start 2023-11-24T06:19:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d1256f700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d0c15e160 0x7f3d0c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d1256f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c0afc90 0x7f3d0c073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d1256f700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3d0c0744d0 0x7f3d0c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d1256f700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0c0b4500 con 0x7f3d0c0744d0 2023-11-24T06:19:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d1256f700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0c074960 con 0x7f3d0c15e160 2023-11-24T06:19:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d1256f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c0b3c60 con 0x7f3d0c0afc90 2023-11-24T06:19:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d10b0c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3d0c0744d0 0x7f3d0c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d10b0c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3d0c0744d0 0x7f3d0c06d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37176/0 (socket says 172.21.15.89:37176) 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d10b0c700 1 -- 172.21.15.89:0/492782330 learned_addr learned my addr 172.21.15.89:0/492782330 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.553+0000 7f3d0b7fe700 1 --2- 172.21.15.89:0/492782330 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c0afc90 0x7f3d0c073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d10b0c700 1 -- 172.21.15.89:0/492782330 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c0afc90 msgr2=0x7f3d0c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d10b0c700 1 --2- 172.21.15.89:0/492782330 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c0afc90 0x7f3d0c073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d10b0c700 1 -- 172.21.15.89:0/492782330 >> v2:172.21.15.123:3300/0 conn(0x7f3d0c15e160 msgr2=0x7f3d0c0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d0bfff700 1 --2- 172.21.15.89:0/492782330 >> v2:172.21.15.123:3300/0 conn(0x7f3d0c15e160 0x7f3d0c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d10b0c700 1 --2- 172.21.15.89:0/492782330 >> v2:172.21.15.123:3300/0 conn(0x7f3d0c15e160 0x7f3d0c0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d10b0c700 1 -- 172.21.15.89:0/492782330 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0b58c0 con 0x7f3d0c0744d0 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d0bfff700 1 --2- 172.21.15.89:0/492782330 >> v2:172.21.15.123:3300/0 conn(0x7f3d0c15e160 0x7f3d0c0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d10b0c700 1 --2- 172.21.15.89:0/492782330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c06d930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c09efa0 tx=0x7f3d0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7db555602150a5a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d0affd700 1 -- 172.21.15.89:0/492782330 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0400e070 con 0x7f3d0c0744d0 2023-11-24T06:19:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d0affd700 1 -- 172.21.15.89:0/492782330 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0400b9c0 con 0x7f3d0c0744d0 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d0affd700 1 -- 172.21.15.89:0/492782330 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0400bd70 con 0x7f3d0c0744d0 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d1256f700 1 -- 172.21.15.89:0/492782330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 msgr2=0x7f3d0c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d1256f700 1 --2- 172.21.15.89:0/492782330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c06d930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c09efa0 tx=0x7f3d0400b330 comp rx=0 tx=0).stop 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d1256f700 1 -- 172.21.15.89:0/492782330 shutdown_connections 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d1256f700 1 --2- 172.21.15.89:0/492782330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c06d930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d1256f700 1 --2- 172.21.15.89:0/492782330 >> v2:172.21.15.123:3300/0 conn(0x7f3d0c15e160 0x7f3d0c0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d1256f700 1 --2- 172.21.15.89:0/492782330 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c0afc90 0x7f3d0c073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.554+0000 7f3d1256f700 1 -- 172.21.15.89:0/492782330 >> 172.21.15.89:0/492782330 conn(0x7f3d0c0d0bf0 msgr2=0x7f3d0c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.555+0000 7f3d1256f700 1 -- 172.21.15.89:0/492782330 shutdown_connections 2023-11-24T06:19:58.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.555+0000 7f3d1256f700 1 -- 172.21.15.89:0/492782330 wait complete. 2023-11-24T06:19:58.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.556+0000 7f3d1256f700 1 Processor -- start 2023-11-24T06:19:58.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.556+0000 7f3d1256f700 1 -- start start 2023-11-24T06:19:58.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d1256f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d1256f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d0c0afc90 0x7f3d0c1f3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d1256f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c15e160 0x7f3d0c153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d1256f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c152ff0 con 0x7f3d0c0744d0 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d1256f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c1f3a70 con 0x7f3d0c0afc90 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d1256f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c15efa0 con 0x7f3d0c15e160 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37178/0 (socket says 172.21.15.89:37178) 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 -- 172.21.15.89:0/4075042113 learned_addr learned my addr 172.21.15.89:0/4075042113 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 -- 172.21.15.89:0/4075042113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c15e160 msgr2=0x7f3d0c153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c15e160 0x7f3d0c153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 -- 172.21.15.89:0/4075042113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d0c0afc90 msgr2=0x7f3d0c1f3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d0c0afc90 0x7f3d0c1f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 -- 172.21.15.89:0/4075042113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d04009b30 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d0bfff700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c155de0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000c7f0 tx=0x7f3d0000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d08ff9700 1 -- 172.21.15.89:0/4075042113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00011800 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d08ff9700 1 -- 172.21.15.89:0/4075042113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d00011960 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0c15f280 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d08ff9700 1 -- 172.21.15.89:0/4075042113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00011d10 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.557+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0c1534e0 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.558+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0c068fb0 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.559+0000 7f3d08ff9700 1 -- 172.21.15.89:0/4075042113 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3d00018360 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.559+0000 7f3d08ff9700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3cf806aed0 0x7f3cf806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.559+0000 7f3d08ff9700 1 -- 172.21.15.89:0/4075042113 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6543+0+0 (secure 0 0 0) 0x7f3d00089830 con 0x7f3d0c0744d0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.559+0000 7f3d0b7fe700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3cf806aed0 0x7f3cf806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.560+0000 7f3d0b7fe700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3cf806aed0 0x7f3cf806d390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c1f7040 tx=0x7f3cfc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.564+0000 7f3d08ff9700 1 -- 172.21.15.89:0/4075042113 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d00054630 con 0x7f3d0c0744d0 2023-11-24T06:19:58.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.754+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool ls", "detail": "detail", "format": "json"} v 0) v1 -- 0x7f3d0c152490 con 0x7f3d0c0744d0 2023-11-24T06:19:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.755+0000 7f3d08ff9700 1 -- 172.21.15.89:0/4075042113 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool ls", "detail": "detail", "format": "json"}]=0 v38) v1 ==== 97+0+9120 (secure 0 0 0) 0x7f3d00057c50 con 0x7f3d0c0744d0 2023-11-24T06:19:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:58.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3cf806aed0 msgr2=0x7f3cf806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3cf806aed0 0x7f3cf806d390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c1f7040 tx=0x7f3cfc008040 comp rx=0 tx=0).stop 2023-11-24T06:19:58.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 msgr2=0x7f3d0c155de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c155de0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000c7f0 tx=0x7f3d0000cbb0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 shutdown_connections 2023-11-24T06:19:58.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0c15e160 0x7f3d0c153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c0744d0 0x7f3d0c155de0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3cf806aed0 0x7f3cf806d390 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 --2- 172.21.15.89:0/4075042113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d0c0afc90 0x7f3d0c1f3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 >> 172.21.15.89:0/4075042113 conn(0x7f3d0c0d0bf0 msgr2=0x7f3d0c15bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:58.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 shutdown_connections 2023-11-24T06:19:58.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.757+0000 7f3d1256f700 1 -- 172.21.15.89:0/4075042113 wait complete. 2023-11-24T06:19:58.772 INFO:teuthology.orchestra.run.smithi089.stdout:[{"pool_id":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool_id":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool_id":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool_id":6,"pool_name":"cephfs.vol3975.meta","create_time":"2023-11-24T06:19:55.111266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"vol3975"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool_id":7,"pool_name":"cephfs.vol3975.data","create_time":"2023-11-24T06:19:56.048405+0000","flags":163841,"flags_names":"hashpspool,creating,bulk","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"vol3975"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}] 2023-11-24T06:19:58.772 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs get vol3975 --format=json 2023-11-24T06:19:58.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.891+0000 7f57d7acc700 1 Processor -- start 2023-11-24T06:19:58.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.891+0000 7f57d7acc700 1 -- start start 2023-11-24T06:19:58.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d7acc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57d015e080 0x7f57d00b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d7acc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00ae430 con 0x7f57d006c710 2023-11-24T06:19:58.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d7acc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00ae710 con 0x7f57d00af4d0 2023-11-24T06:19:58.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d7acc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d00af0b0 con 0x7f57d015e080 2023-11-24T06:19:58.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d5868700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57d015e080 0x7f57d00b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d6069700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f57d006c710 0x7f57d006e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40694/0 (socket says 172.21.15.89:40694) 2023-11-24T06:19:58.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d6069700 1 -- 172.21.15.89:0/1514219451 learned_addr learned my addr 172.21.15.89:0/1514219451 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d4866700 1 -- 172.21.15.89:0/1514219451 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811260345 0 0) 0x7f57d00ae430 con 0x7f57d006c710 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d4866700 1 -- 172.21.15.89:0/1514219451 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57bc0035b0 con 0x7f57d006c710 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d5868700 1 -- 172.21.15.89:0/1514219451 >> v1:172.21.15.123:6789/0 conn(0x7f57d00af4d0 legacy=0x7f57d006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d5868700 1 -- 172.21.15.89:0/1514219451 >> v1:172.21.15.89:6789/0 conn(0x7f57d006c710 legacy=0x7f57d006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d5868700 1 -- 172.21.15.89:0/1514219451 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d00ae9f0 con 0x7f57d015e080 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.892+0000 7f57d5868700 1 --2- 172.21.15.89:0/1514219451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d00b5050 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009840 tx=0x7f57cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90dac0afcfd82301 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d4866700 1 -- 172.21.15.89:0/1514219451 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc011830 con 0x7f57d015e080 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d4866700 1 -- 172.21.15.89:0/1514219451 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc011990 con 0x7f57d015e080 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d4866700 1 -- 172.21.15.89:0/1514219451 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc011d00 con 0x7f57d015e080 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d7acc700 1 -- 172.21.15.89:0/1514219451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 msgr2=0x7f57d00b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d7acc700 1 --2- 172.21.15.89:0/1514219451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d00b5050 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009840 tx=0x7f57cc009b50 comp rx=0 tx=0).stop 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d7acc700 1 -- 172.21.15.89:0/1514219451 shutdown_connections 2023-11-24T06:19:58.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d7acc700 1 --2- 172.21.15.89:0/1514219451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d00b5050 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d7acc700 1 -- 172.21.15.89:0/1514219451 >> 172.21.15.89:0/1514219451 conn(0x7f57d0071b90 msgr2=0x7f57d0071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:58.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d7acc700 1 -- 172.21.15.89:0/1514219451 shutdown_connections 2023-11-24T06:19:58.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.893+0000 7f57d7acc700 1 -- 172.21.15.89:0/1514219451 wait complete. 2023-11-24T06:19:58.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.894+0000 7f57d7acc700 1 Processor -- start 2023-11-24T06:19:58.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.894+0000 7f57d7acc700 1 -- start start 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d7acc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57d006c710 0x7f57d014a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d7acc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57d00af4d0 0x7f57d0154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d7acc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d0155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d7acc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0157ff0 con 0x7f57d015e080 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d7acc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0155070 con 0x7f57d00af4d0 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d7acc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0155350 con 0x7f57d006c710 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d6069700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d0155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d6069700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d0155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37196/0 (socket says 172.21.15.89:37196) 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d6069700 1 -- 172.21.15.89:0/1220558448 learned_addr learned my addr 172.21.15.89:0/1220558448 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d5868700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57d006c710 0x7f57d014a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d5067700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57d00af4d0 0x7f57d0154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d6069700 1 -- 172.21.15.89:0/1220558448 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57d006c710 msgr2=0x7f57d014a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d6069700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57d006c710 0x7f57d014a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d6069700 1 -- 172.21.15.89:0/1220558448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57d00af4d0 msgr2=0x7f57d0154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d6069700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57d00af4d0 0x7f57d0154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.895+0000 7f57d6069700 1 -- 172.21.15.89:0/1220558448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc0094a0 con 0x7f57d015e080 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.896+0000 7f57d6069700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d0155f40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f57c800cbb0 tx=0x7f57c800cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.896+0000 7f57c67fc700 1 -- 172.21.15.89:0/1220558448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c800a9e0 con 0x7f57d015e080 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.896+0000 7f57c67fc700 1 -- 172.21.15.89:0/1220558448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c800ab40 con 0x7f57d015e080 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.896+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d0155630 con 0x7f57d015e080 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.896+0000 7f57c67fc700 1 -- 172.21.15.89:0/1220558448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c8012400 con 0x7f57d015e080 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.896+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d0154c40 con 0x7f57d015e080 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.897+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d00753f0 con 0x7f57d015e080 2023-11-24T06:19:58.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.897+0000 7f57c67fc700 1 -- 172.21.15.89:0/1220558448 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f57c801b3b0 con 0x7f57d015e080 2023-11-24T06:19:58.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.897+0000 7f57c67fc700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f57b406aed0 0x7f57b406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:58.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.898+0000 7f57c67fc700 1 -- 172.21.15.89:0/1220558448 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6543+0+0 (secure 0 0 0) 0x7f57c8089450 con 0x7f57d015e080 2023-11-24T06:19:58.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.898+0000 7f57d5868700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f57b406aed0 0x7f57b406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:58.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.898+0000 7f57d5868700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f57b406aed0 0x7f57b406d390 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f57cc000c00 tx=0x7f57cc01c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:58.902+0000 7f57c67fc700 1 -- 172.21.15.89:0/1220558448 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57c8054300 con 0x7f57d015e080 2023-11-24T06:19:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.140+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs get", "fs_name": "vol3975", "format": "json"} v 0) v1 -- 0x7f57d0067f60 con 0x7f57d015e080 2023-11-24T06:19:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.141+0000 7f57c67fc700 1 -- 172.21.15.89:0/1220558448 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs get", "fs_name": "vol3975", "format": "json"}]=0 v20) v1 ==== 94+0+2008 (secure 0 0 0) 0x7f57c8057920 con 0x7f57d015e080 2023-11-24T06:19:59.144 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:19:59.144 INFO:teuthology.orchestra.run.smithi089.stdout:{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:19:57.031384+0000","modified":"2023-11-24T06:19:58.043300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/2053513777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2053513777},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053513777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"vol3975","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":3} 2023-11-24T06:19:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f57b406aed0 msgr2=0x7f57b406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f57b406aed0 0x7f57b406d390 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f57cc000c00 tx=0x7f57cc01c040 comp rx=0 tx=0).stop 2023-11-24T06:19:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 msgr2=0x7f57d0155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d0155f40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f57c800cbb0 tx=0x7f57c800cec0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 shutdown_connections 2023-11-24T06:19:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57d015e080 0x7f57d0155f40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57d006c710 0x7f57d014a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f57b406aed0 0x7f57b406d390 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 --2- 172.21.15.89:0/1220558448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57d00af4d0 0x7f57d0154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 >> 172.21.15.89:0/1220558448 conn(0x7f57d0071b90 msgr2=0x7f57d0160470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 shutdown_connections 2023-11-24T06:19:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.143+0000 7f57d7acc700 1 -- 172.21.15.89:0/1220558448 wait complete. 2023-11-24T06:19:59.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm vol3975 --yes-i-really-mean-it 2023-11-24T06:19:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.278+0000 7fac11e3f700 1 Processor -- start 2023-11-24T06:19:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.278+0000 7fac11e3f700 1 -- start start 2023-11-24T06:19:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac11e3f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c0d0230 0x7fac0c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac11e3f700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c0b2880 con 0x7fac0c0b24a0 2023-11-24T06:19:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac11e3f700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c0b31a0 con 0x7fac0c0b2ab0 2023-11-24T06:19:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac11e3f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0d0c40 con 0x7fac0c0d0230 2023-11-24T06:19:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0b7fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fac0c0b24a0 0x7fac0c0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40704/0 (socket says 172.21.15.89:40704) 2023-11-24T06:19:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0b7fe700 1 -- 172.21.15.89:0/2062029415 learned_addr learned my addr 172.21.15.89:0/2062029415 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:19:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0bfff700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c0d0230 0x7fac0c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568879629 0 0) 0x7fac0c0b2880 con 0x7fac0c0b24a0 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf80035b0 con 0x7fac0c0b24a0 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4088409941 0 0) 0x7fabf80035b0 con 0x7fac0c0b24a0 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c0d0230 msgr2=0x7fac0c0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c0d0230 0x7fac0c0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 >> v1:172.21.15.123:6789/0 conn(0x7fac0c0b2ab0 legacy=0x7fac0c0cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c0b3480 con 0x7fac0c0b24a0 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fabfc001670 con 0x7fac0c0b24a0 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.279+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 >> v1:172.21.15.89:6789/0 conn(0x7fac0c0b24a0 legacy=0x7fac0c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:59.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf8003a30 0x7fabf8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c0d0230 0x7fabf8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf8004d90 0x7fabf8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0d0c40 con 0x7fabf8004d90 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf800bbd0 con 0x7fabf8003a30 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf800bd90 con 0x7fac0c0d0230 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0bfff700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf8004d90 0x7fabf8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0b7fe700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf8003a30 0x7fabf8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac02ffd700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c0d0230 0x7fabf8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0bfff700 1 -- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c0d0230 msgr2=0x7fabf8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0bfff700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c0d0230 0x7fabf8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0bfff700 1 -- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf8003a30 msgr2=0x7fabf8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:19:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0bfff700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf8003a30 0x7fabf8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0bfff700 1 -- 172.21.15.89:0/2062029415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80035b0 con 0x7fabf8004d90 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0bfff700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf8004d90 0x7fabf8005210 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac04009fc0 tx=0x7fabf8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d4f77fc8377d21 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0400ec30 con 0x7fabf8004d90 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0400f040 con 0x7fabf8004d90 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.280+0000 7fac0affd700 1 -- 172.21.15.89:0/2062029415 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac04014480 con 0x7fabf8004d90 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.281+0000 7fac11e3f700 1 -- 172.21.15.89:0/2062029415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf8004d90 msgr2=0x7fabf8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.281+0000 7fac11e3f700 1 --2- 172.21.15.89:0/2062029415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf8004d90 0x7fabf8005210 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac04009fc0 tx=0x7fabf8005880 comp rx=0 tx=0).stop 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.281+0000 7fac0bfff700 1 -- 172.21.15.89:0/2062029415 reap_dead start 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.281+0000 7fac11e3f700 1 -- 172.21.15.89:0/2062029415 shutdown_connections 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.281+0000 7fac11e3f700 1 -- 172.21.15.89:0/2062029415 >> 172.21.15.89:0/2062029415 conn(0x7fac0c0b3810 msgr2=0x7fac0c0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.281+0000 7fac11e3f700 1 -- 172.21.15.89:0/2062029415 shutdown_connections 2023-11-24T06:19:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.281+0000 7fac11e3f700 1 -- 172.21.15.89:0/2062029415 wait complete. 2023-11-24T06:19:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.282+0000 7fac11e3f700 1 Processor -- start 2023-11-24T06:19:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.282+0000 7fac11e3f700 1 -- start start 2023-11-24T06:19:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac11e3f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c152030 0x7fac0c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac11e3f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac0c0c8320 0x7fac0c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac11e3f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0c06f3d0 0x7fac0c06f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac11e3f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c06fd90 con 0x7fac0c06f3d0 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac11e3f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0b5c10 con 0x7fac0c0c8320 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac11e3f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0c87a0 con 0x7fac0c152030 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0c06f3d0 0x7fac0c06f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0c06f3d0 0x7fac0c06f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37216/0 (socket says 172.21.15.89:37216) 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 -- 172.21.15.89:0/2496731179 learned_addr learned my addr 172.21.15.89:0/2496731179 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 -- 172.21.15.89:0/2496731179 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c152030 msgr2=0x7fac0c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c152030 0x7fac0c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 -- 172.21.15.89:0/2496731179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac0c0c8320 msgr2=0x7fac0c0b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac0c0c8320 0x7fac0c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 -- 172.21.15.89:0/2496731179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c0b65e0 con 0x7fac0c06f3d0 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.283+0000 7fac0bfff700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0c06f3d0 0x7fac0c06f850 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fabf800bba0 tx=0x7fac04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.284+0000 7fac08ff9700 1 -- 172.21.15.89:0/2496731179 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac04007000 con 0x7fac0c06f3d0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.284+0000 7fac08ff9700 1 -- 172.21.15.89:0/2496731179 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac04015b30 con 0x7fac0c06f3d0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.284+0000 7fac08ff9700 1 -- 172.21.15.89:0/2496731179 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0401d420 con 0x7fac0c06f3d0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.284+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0c0b5000 con 0x7fac0c06f3d0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.284+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0c0b55c0 con 0x7fac0c06f3d0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.285+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0c068fb0 con 0x7fac0c06f3d0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.285+0000 7fac08ff9700 1 -- 172.21.15.89:0/2496731179 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fac0401d580 con 0x7fac0c06f3d0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.285+0000 7fac08ff9700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fabec06aed0 0x7fabec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.286+0000 7fac08ff9700 1 -- 172.21.15.89:0/2496731179 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6543+0+0 (secure 0 0 0) 0x7fac0408b220 con 0x7fac0c06f3d0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.286+0000 7fac0b7fe700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fabec06aed0 0x7fabec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:19:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.286+0000 7fac0b7fe700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fabec06aed0 0x7fabec06d390 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fabfc001c90 tx=0x7fabfc001b20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:19:59.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.290+0000 7fac08ff9700 1 -- 172.21.15.89:0/2496731179 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac04056140 con 0x7fac0c06f3d0 2023-11-24T06:19:59.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:19:59.509+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "vol3975", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fac0c161cf0 con 0x7fabec06aed0 2023-11-24T06:19:59.516 INFO:tasks.ceph.mon.a.smithi089.stderr:2023-11-24T06:19:59.513+0000 7f7eccc4c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T06:19:59.545 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-11-24T06:20:00.001 INFO:tasks.ceph.mon.a.smithi089.stderr:2023-11-24T06:19:59.999+0000 7f7ecf451700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2023-11-24T06:20:02.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.540+0000 7fac08ff9700 1 -- 172.21.15.89:0/2496731179 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+78 (secure 0 0 0) 0x7fac0c161cf0 con 0x7fabec06aed0 2023-11-24T06:20:02.544 INFO:teuthology.orchestra.run.smithi089.stdout:metadata pool: cephfs.vol3975.meta data pool: ['cephfs.vol3975.data'] removed 2023-11-24T06:20:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fabec06aed0 msgr2=0x7fabec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fabec06aed0 0x7fabec06d390 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fabfc001c90 tx=0x7fabfc001b20 comp rx=0 tx=0).stop 2023-11-24T06:20:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0c06f3d0 msgr2=0x7fac0c06f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0c06f3d0 0x7fac0c06f850 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fabf800bba0 tx=0x7fac04009750 comp rx=0 tx=0).stop 2023-11-24T06:20:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 shutdown_connections 2023-11-24T06:20:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0c06f3d0 0x7fac0c06f850 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac0c152030 0x7fac0c0b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fabec06aed0 0x7fabec06d390 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 --2- 172.21.15.89:0/2496731179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac0c0c8320 0x7fac0c0b60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 >> 172.21.15.89:0/2496731179 conn(0x7fac0c0b3810 msgr2=0x7fac0c0cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:02.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 shutdown_connections 2023-11-24T06:20:02.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.544+0000 7fac11e3f700 1 -- 172.21.15.89:0/2496731179 wait complete. 2023-11-24T06:20:02.562 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T06:20:02.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:20:02.614 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:20:02.665 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T06:20:02.666 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:02.666 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:02.684 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:02.684 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:02.745 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.905+0000 7fa7ea9ea700 1 Processor -- start 2023-11-24T06:20:02.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.905+0000 7fa7ea9ea700 1 -- start start 2023-11-24T06:20:02.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7ea9ea700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa7e415a3b0 0x7fa7e40b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:02.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7ea9ea700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7e40cf360 0x7fa7e40b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:02.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7ea9ea700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:02.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7ea9ea700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e40cf8b0 con 0x7fa7e40cf360 2023-11-24T06:20:02.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7ea9ea700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e415a790 con 0x7fa7e415a3b0 2023-11-24T06:20:02.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7ea9ea700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40b4c40 con 0x7fa7e40b3da0 2023-11-24T06:20:02.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7e37fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7e40cf360 0x7fa7e40b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:02.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7e37fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7e40cf360 0x7fa7e40b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37234/0 (socket says 172.21.15.89:37234) 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7e37fe700 1 -- 172.21.15.89:0/2069801613 learned_addr learned my addr 172.21.15.89:0/2069801613 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7e37fe700 1 -- 172.21.15.89:0/2069801613 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 msgr2=0x7fa7e406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7e8f87700 1 --2- 172.21.15.89:0/2069801613 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7e3fff700 1 --2- 172.21.15.89:0/2069801613 >> v2:172.21.15.123:3300/0 conn(0x7fa7e415a3b0 0x7fa7e40b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7e37fe700 1 --2- 172.21.15.89:0/2069801613 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e406dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.906+0000 7fa7e37fe700 1 -- 172.21.15.89:0/2069801613 >> v2:172.21.15.123:3300/0 conn(0x7fa7e415a3b0 msgr2=0x7fa7e40b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7e37fe700 1 --2- 172.21.15.89:0/2069801613 >> v2:172.21.15.123:3300/0 conn(0x7fa7e415a3b0 0x7fa7e40b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7e37fe700 1 -- 172.21.15.89:0/2069801613 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e4070c60 con 0x7fa7e40cf360 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7e3fff700 1 --2- 172.21.15.89:0/2069801613 >> v2:172.21.15.123:3300/0 conn(0x7fa7e415a3b0 0x7fa7e40b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7e37fe700 1 --2- 172.21.15.89:0/2069801613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40b3860 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa7e404fca0 tx=0x7fa7d400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2604cda972a1d9cb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7e2ffd700 1 -- 172.21.15.89:0/2069801613 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d400dd10 con 0x7fa7e40cf360 2023-11-24T06:20:02.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7e2ffd700 1 -- 172.21.15.89:0/2069801613 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d400de70 con 0x7fa7e40cf360 2023-11-24T06:20:02.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7e2ffd700 1 -- 172.21.15.89:0/2069801613 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4010670 con 0x7fa7e40cf360 2023-11-24T06:20:02.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2069801613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 msgr2=0x7fa7e40b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:02.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2069801613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40b3860 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa7e404fca0 tx=0x7fa7d400bea0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2069801613 shutdown_connections 2023-11-24T06:20:02.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2069801613 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e406dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2069801613 >> v2:172.21.15.123:3300/0 conn(0x7fa7e415a3b0 0x7fa7e40b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2069801613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40b3860 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2069801613 >> 172.21.15.89:0/2069801613 conn(0x7fa7e40b1740 msgr2=0x7fa7e40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:02.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2069801613 shutdown_connections 2023-11-24T06:20:02.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.907+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2069801613 wait complete. 2023-11-24T06:20:02.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.909+0000 7fa7ea9ea700 1 Processor -- start 2023-11-24T06:20:02.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.909+0000 7fa7ea9ea700 1 -- start start 2023-11-24T06:20:02.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.909+0000 7fa7ea9ea700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e41f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:02.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7ea9ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:02.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7ea9ea700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e415a3b0 0x7fa7e40ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:02.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7ea9ea700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40c9e20 con 0x7fa7e40cf360 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7ea9ea700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40c92a0 con 0x7fa7e415a3b0 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7ea9ea700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40c9580 con 0x7fa7e40b3da0 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37240/0 (socket says 172.21.15.89:37240) 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 -- 172.21.15.89:0/2966592372 learned_addr learned my addr 172.21.15.89:0/2966592372 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e3fff700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e41f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e8f87700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e415a3b0 0x7fa7e40ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 -- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 msgr2=0x7fa7e41f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e41f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 -- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e415a3b0 msgr2=0x7fa7e40ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e415a3b0 0x7fa7e40ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 -- 172.21.15.89:0/2966592372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d400b7f0 con 0x7fa7e40cf360 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e3fff700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e41f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e37fe700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40cdef0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400bb90 tx=0x7fa7d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e0ff9700 1 -- 172.21.15.89:0/2966592372 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4003da0 con 0x7fa7e40cf360 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e0ff9700 1 -- 172.21.15.89:0/2966592372 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d4010470 con 0x7fa7e40cf360 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e40c9860 con 0x7fa7e40cf360 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7e0ff9700 1 -- 172.21.15.89:0/2966592372 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4010800 con 0x7fa7e40cf360 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.910+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e41f39c0 con 0x7fa7e40cf360 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.911+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e41f3b50 con 0x7fa7e40cf360 2023-11-24T06:20:02.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.912+0000 7fa7e0ff9700 1 -- 172.21.15.89:0/2966592372 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa7d4019400 con 0x7fa7e40cf360 2023-11-24T06:20:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.912+0000 7fa7e0ff9700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7d006aed0 0x7fa7d006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.912+0000 7fa7e0ff9700 1 -- 172.21.15.89:0/2966592372 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa7d408add0 con 0x7fa7e40cf360 2023-11-24T06:20:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.912+0000 7fa7e3fff700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7d006aed0 0x7fa7d006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.913+0000 7fa7e3fff700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7d006aed0 0x7fa7d006d390 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006fd0 tx=0x7fa7d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:02.916+0000 7fa7e0ff9700 1 -- 172.21.15.89:0/2966592372 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7d4055eb0 con 0x7fa7e40cf360 2023-11-24T06:20:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.108+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7e40751e0 con 0x7fa7e40cf360 2023-11-24T06:20:03.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.109+0000 7fa7e0ff9700 1 -- 172.21.15.89:0/2966592372 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16662 (secure 0 0 0) 0x7fa7d40594d0 con 0x7fa7e40cf360 2023-11-24T06:20:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:03.113 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:02.530677+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/2053513777":"2023-11-25T06:19:59.514768+0000","172.21.15.89:6835/2053513777":"2023-11-25T06:19:59.514768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:03.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.111+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7d006aed0 msgr2=0x7fa7d006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.111+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7d006aed0 0x7fa7d006d390 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006fd0 tx=0x7fa7d8008040 comp rx=0 tx=0).stop 2023-11-24T06:20:03.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.111+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 msgr2=0x7fa7e40cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.111+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40cdef0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400bb90 tx=0x7fa7d4004750 comp rx=0 tx=0).stop 2023-11-24T06:20:03.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.112+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 shutdown_connections 2023-11-24T06:20:03.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.112+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7d006aed0 0x7fa7d006d390 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.112+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e415a3b0 0x7fa7e40ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.112+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7e40b3da0 0x7fa7e41f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.112+0000 7fa7ea9ea700 1 --2- 172.21.15.89:0/2966592372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7e40cf360 0x7fa7e40cdef0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.112+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 >> 172.21.15.89:0/2966592372 conn(0x7fa7e40b1740 msgr2=0x7fa7e40b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:03.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.112+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 shutdown_connections 2023-11-24T06:20:03.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.112+0000 7fa7ea9ea700 1 -- 172.21.15.89:0/2966592372 wait complete. 2023-11-24T06:20:03.127 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:19:41.783017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:19:42.953803+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:03.127 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:03.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.248+0000 7f107eaea700 1 Processor -- start 2023-11-24T06:20:03.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.248+0000 7f107eaea700 1 -- start start 2023-11-24T06:20:03.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f107eaea700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f107815b370 0x7f107815b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f107eaea700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780ae5e0 con 0x7f10780b3c60 2023-11-24T06:20:03.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f107eaea700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107815be30 con 0x7f10781587f0 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f107eaea700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107815c110 con 0x7f107815b370 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f1077fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f107815b370 0x7f107815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f107d087700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f10780b3c60 0x7f10780af060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40754/0 (socket says 172.21.15.89:40754) 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f107d087700 1 -- 172.21.15.89:0/503798244 learned_addr learned my addr 172.21.15.89:0/503798244 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f1077fff700 1 -- 172.21.15.89:0/503798244 >> v1:172.21.15.123:6789/0 conn(0x7f10781587f0 legacy=0x7f107806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f1077fff700 1 -- 172.21.15.89:0/503798244 >> v1:172.21.15.89:6789/0 conn(0x7f10780b3c60 legacy=0x7f10780af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f10777fe700 1 -- 172.21.15.89:0/503798244 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162479219 0 0) 0x7f10780ae5e0 con 0x7f10780b3c60 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f1077fff700 1 -- 172.21.15.89:0/503798244 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10780ae860 con 0x7f107815b370 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.249+0000 7f1077fff700 1 --2- 172.21.15.89:0/503798244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f107815b800 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1064009870 tx=0x7f1064009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a3ff2944fbbb4f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f10777fe700 1 -- 172.21.15.89:0/503798244 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064004230 con 0x7f107815b370 2023-11-24T06:20:03.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f10777fe700 1 -- 172.21.15.89:0/503798244 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1064004390 con 0x7f107815b370 2023-11-24T06:20:03.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f10777fe700 1 -- 172.21.15.89:0/503798244 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10640036a0 con 0x7f107815b370 2023-11-24T06:20:03.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f107eaea700 1 -- 172.21.15.89:0/503798244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 msgr2=0x7f107815b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f107eaea700 1 --2- 172.21.15.89:0/503798244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f107815b800 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1064009870 tx=0x7f1064009b80 comp rx=0 tx=0).stop 2023-11-24T06:20:03.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f107eaea700 1 -- 172.21.15.89:0/503798244 shutdown_connections 2023-11-24T06:20:03.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f107eaea700 1 --2- 172.21.15.89:0/503798244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f107815b800 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f107eaea700 1 -- 172.21.15.89:0/503798244 >> 172.21.15.89:0/503798244 conn(0x7f10780d0bf0 msgr2=0x7f10780d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:03.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f107eaea700 1 -- 172.21.15.89:0/503798244 shutdown_connections 2023-11-24T06:20:03.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.250+0000 7f107eaea700 1 -- 172.21.15.89:0/503798244 wait complete. 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.251+0000 7f107eaea700 1 Processor -- start 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.251+0000 7f107eaea700 1 -- start start 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107eaea700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10780b3c60 0x7f1078148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107eaea700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10781587f0 0x7f10781f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107eaea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f10781462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107eaea700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10781f50d0 con 0x7f107815b370 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107eaea700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1078146cb0 con 0x7f10781587f0 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107eaea700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1078149b20 con 0x7f10780b3c60 2023-11-24T06:20:03.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107d087700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f10781462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107d087700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f10781462a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37256/0 (socket says 172.21.15.89:37256) 2023-11-24T06:20:03.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.252+0000 7f107d087700 1 -- 172.21.15.89:0/4225606413 learned_addr learned my addr 172.21.15.89:0/4225606413 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:03.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107c886700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10780b3c60 0x7f1078148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f1077fff700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10781587f0 0x7f10781f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107d087700 1 -- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10780b3c60 msgr2=0x7f1078148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107d087700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10780b3c60 0x7f1078148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107d087700 1 -- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10781587f0 msgr2=0x7f10781f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107d087700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10781587f0 0x7f10781f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107d087700 1 -- 172.21.15.89:0/4225606413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10640094a0 con 0x7f107815b370 2023-11-24T06:20:03.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107c886700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10780b3c60 0x7f1078148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:03.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f1077fff700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10781587f0 0x7f10781f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:03.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107d087700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f10781462a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1070002210 tx=0x7f107000c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f10757fa700 1 -- 172.21.15.89:0/4225606413 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10700070d0 con 0x7f107815b370 2023-11-24T06:20:03.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f10757fa700 1 -- 172.21.15.89:0/4225606413 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1070007230 con 0x7f107815b370 2023-11-24T06:20:03.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1078149280 con 0x7f107815b370 2023-11-24T06:20:03.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f10757fa700 1 -- 172.21.15.89:0/4225606413 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10700075c0 con 0x7f107815b370 2023-11-24T06:20:03.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.253+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1078149840 con 0x7f107815b370 2023-11-24T06:20:03.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.254+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1078146e40 con 0x7f107815b370 2023-11-24T06:20:03.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.254+0000 7f10757fa700 1 -- 172.21.15.89:0/4225606413 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1070007720 con 0x7f107815b370 2023-11-24T06:20:03.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.255+0000 7f10757fa700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f105c06aed0 0x7f105c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.255+0000 7f10757fa700 1 -- 172.21.15.89:0/4225606413 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f107008c680 con 0x7f107815b370 2023-11-24T06:20:03.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.255+0000 7f107c886700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f105c06aed0 0x7f105c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.255+0000 7f107c886700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f105c06aed0 0x7f105c06d390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f106c003f40 tx=0x7f106c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.259+0000 7f10757fa700 1 -- 172.21.15.89:0/4225606413 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10700576d0 con 0x7f107815b370 2023-11-24T06:20:03.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.451+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1078068fb0 con 0x7f107815b370 2023-11-24T06:20:03.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.451+0000 7f10757fa700 1 -- 172.21.15.89:0/4225606413 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10700079d0 con 0x7f107815b370 2023-11-24T06:20:03.454 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:03.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f105c06aed0 msgr2=0x7f105c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f105c06aed0 0x7f105c06d390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f106c003f40 tx=0x7f106c009040 comp rx=0 tx=0).stop 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 msgr2=0x7f10781462a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f10781462a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1070002210 tx=0x7f107000c550 comp rx=0 tx=0).stop 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 shutdown_connections 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f107815b370 0x7f10781462a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10780b3c60 0x7f1078148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f105c06aed0 0x7f105c06d390 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 --2- 172.21.15.89:0/4225606413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10781587f0 0x7f10781f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.453+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 >> 172.21.15.89:0/4225606413 conn(0x7f10780d0bf0 msgr2=0x7f107806f250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.454+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 shutdown_connections 2023-11-24T06:20:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.454+0000 7f107eaea700 1 -- 172.21.15.89:0/4225606413 wait complete. 2023-11-24T06:20:03.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:20:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.590+0000 7ffb5c73d700 1 Processor -- start 2023-11-24T06:20:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.590+0000 7ffb5c73d700 1 -- start start 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5c73d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb5415a620 0x7ffb540ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5c73d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb540cc540 0x7ffb5406e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5c73d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffb5406eb20 0x7ffb5406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5c73d700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5406efb0 con 0x7ffb5415a620 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5c73d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb540b2f20 con 0x7ffb5406eb20 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5c73d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb540ce690 con 0x7ffb540cc540 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5a4d9700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb5415a620 0x7ffb540ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5a4d9700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb5415a620 0x7ffb540ce150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37260/0 (socket says 172.21.15.89:37260) 2023-11-24T06:20:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5a4d9700 1 -- 172.21.15.89:0/3937067694 learned_addr learned my addr 172.21.15.89:0/3937067694 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb59cd8700 1 --2- 172.21.15.89:0/3937067694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb540cc540 0x7ffb5406e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5acda700 1 --2- 172.21.15.89:0/3937067694 >> v2:172.21.15.123:3300/0 conn(0x7ffb5406eb20 0x7ffb5406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5a4d9700 1 -- 172.21.15.89:0/3937067694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb540cc540 msgr2=0x7ffb5406e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5a4d9700 1 --2- 172.21.15.89:0/3937067694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb540cc540 0x7ffb5406e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5a4d9700 1 -- 172.21.15.89:0/3937067694 >> v2:172.21.15.123:3300/0 conn(0x7ffb5406eb20 msgr2=0x7ffb5406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5a4d9700 1 --2- 172.21.15.89:0/3937067694 >> v2:172.21.15.123:3300/0 conn(0x7ffb5406eb20 0x7ffb5406d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5a4d9700 1 -- 172.21.15.89:0/3937067694 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5415aa00 con 0x7ffb5415a620 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.591+0000 7ffb5acda700 1 --2- 172.21.15.89:0/3937067694 >> v2:172.21.15.123:3300/0 conn(0x7ffb5406eb20 0x7ffb5406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5a4d9700 1 --2- 172.21.15.89:0/3937067694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5415a620 0x7ffb540ce150 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ffb540aff60 tx=0x7ffb44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc61cd1400395234 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb594d7700 1 -- 172.21.15.89:0/3937067694 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4400b870 con 0x7ffb5415a620 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb594d7700 1 -- 172.21.15.89:0/3937067694 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4400b9d0 con 0x7ffb5415a620 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb594d7700 1 -- 172.21.15.89:0/3937067694 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4400bd60 con 0x7ffb5415a620 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 -- 172.21.15.89:0/3937067694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5415a620 msgr2=0x7ffb540ce150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/3937067694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5415a620 0x7ffb540ce150 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ffb540aff60 tx=0x7ffb44009bd0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 -- 172.21.15.89:0/3937067694 shutdown_connections 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/3937067694 >> v2:172.21.15.123:3300/0 conn(0x7ffb5406eb20 0x7ffb5406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/3937067694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5415a620 0x7ffb540ce150 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/3937067694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb540cc540 0x7ffb5406e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 -- 172.21.15.89:0/3937067694 >> 172.21.15.89:0/3937067694 conn(0x7ffb540b2050 msgr2=0x7ffb540b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 -- 172.21.15.89:0/3937067694 shutdown_connections 2023-11-24T06:20:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.592+0000 7ffb5c73d700 1 -- 172.21.15.89:0/3937067694 wait complete. 2023-11-24T06:20:03.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.593+0000 7ffb5c73d700 1 Processor -- start 2023-11-24T06:20:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5c73d700 1 -- start start 2023-11-24T06:20:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5c73d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5406eb20 0x7ffb5414cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5c73d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb540cc540 0x7ffb54148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5c73d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb5415a620 0x7ffb5414f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5c73d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb54144e60 con 0x7ffb5406eb20 2023-11-24T06:20:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5c73d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb54144b80 con 0x7ffb540cc540 2023-11-24T06:20:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5c73d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb54148d70 con 0x7ffb5415a620 2023-11-24T06:20:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5a4d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5406eb20 0x7ffb5414cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5a4d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5406eb20 0x7ffb5414cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37264/0 (socket says 172.21.15.89:37264) 2023-11-24T06:20:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5a4d9700 1 -- 172.21.15.89:0/2494754357 learned_addr learned my addr 172.21.15.89:0/2494754357 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb59cd8700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb540cc540 0x7ffb54148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.594+0000 7ffb5acda700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb5415a620 0x7ffb5414f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5a4d9700 1 -- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb5415a620 msgr2=0x7ffb5414f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5a4d9700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb5415a620 0x7ffb5414f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5a4d9700 1 -- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb540cc540 msgr2=0x7ffb54148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5a4d9700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb540cc540 0x7ffb54148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5a4d9700 1 -- 172.21.15.89:0/2494754357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb44009550 con 0x7ffb5406eb20 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5acda700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb5415a620 0x7ffb5414f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5a4d9700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5406eb20 0x7ffb5414cd90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffb44000c00 tx=0x7ffb44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb4affd700 1 -- 172.21.15.89:0/2494754357 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44004540 con 0x7ffb5406eb20 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb4affd700 1 -- 172.21.15.89:0/2494754357 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb440036a0 con 0x7ffb5406eb20 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb54149050 con 0x7ffb5406eb20 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb4affd700 1 -- 172.21.15.89:0/2494754357 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44003a50 con 0x7ffb5406eb20 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.595+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb54149330 con 0x7ffb5406eb20 2023-11-24T06:20:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.596+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb54068d90 con 0x7ffb5406eb20 2023-11-24T06:20:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.596+0000 7ffb4affd700 1 -- 172.21.15.89:0/2494754357 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffb44021070 con 0x7ffb5406eb20 2023-11-24T06:20:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.597+0000 7ffb4affd700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffb3806aed0 0x7ffb3806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.597+0000 7ffb4affd700 1 -- 172.21.15.89:0/2494754357 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffb4408ac80 con 0x7ffb5406eb20 2023-11-24T06:20:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.597+0000 7ffb59cd8700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffb3806aed0 0x7ffb3806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.597+0000 7ffb59cd8700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffb3806aed0 0x7ffb3806d390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffb5415dc40 tx=0x7ffb50008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.601+0000 7ffb4affd700 1 -- 172.21.15.89:0/2494754357 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb44055d60 con 0x7ffb5406eb20 2023-11-24T06:20:03.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.789+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffb54068fb0 con 0x7ffb5406eb20 2023-11-24T06:20:03.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.789+0000 7ffb4affd700 1 -- 172.21.15.89:0/2494754357 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb44015020 con 0x7ffb5406eb20 2023-11-24T06:20:03.792 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:20:03.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffb3806aed0 msgr2=0x7ffb3806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffb3806aed0 0x7ffb3806d390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffb5415dc40 tx=0x7ffb50008040 comp rx=0 tx=0).stop 2023-11-24T06:20:03.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5406eb20 msgr2=0x7ffb5414cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5406eb20 0x7ffb5414cd90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffb44000c00 tx=0x7ffb44004750 comp rx=0 tx=0).stop 2023-11-24T06:20:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 shutdown_connections 2023-11-24T06:20:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffb5415a620 0x7ffb5414f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb5406eb20 0x7ffb5414cd90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffb3806aed0 0x7ffb3806d390 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 --2- 172.21.15.89:0/2494754357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb540cc540 0x7ffb54148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 >> 172.21.15.89:0/2494754357 conn(0x7ffb540b2050 msgr2=0x7ffb540b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 shutdown_connections 2023-11-24T06:20:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.791+0000 7ffb5c73d700 1 -- 172.21.15.89:0/2494754357 wait complete. 2023-11-24T06:20:03.806 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:20:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.927+0000 7f3941387700 1 Processor -- start 2023-11-24T06:20:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.927+0000 7f3941387700 1 -- start start 2023-11-24T06:20:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3941387700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c15e090 0x7f393c0d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3941387700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f393c0d24e0 0x7f393c071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3941387700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393c071f70 con 0x7f393c071b50 2023-11-24T06:20:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3941387700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c0d2a60 con 0x7f393c0d24e0 2023-11-24T06:20:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3941387700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0d2d40 con 0x7f393c15e090 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f393b7fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f393c071b50 0x7f393c0b5920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40778/0 (socket says 172.21.15.89:40778) 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f393b7fe700 1 -- 172.21.15.89:0/3476388355 learned_addr learned my addr 172.21.15.89:0/3476388355 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f393affd700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c15e090 0x7f393c0d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f393a7fc700 1 --2- 172.21.15.89:0/3476388355 >> v2:172.21.15.123:3300/0 conn(0x7f393c0d24e0 0x7f393c071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974937209 0 0) 0x7f393c071f70 con 0x7f393c071b50 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39240035b0 con 0x7f393c071b50 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225836937 0 0) 0x7f39240035b0 con 0x7f393c071b50 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c15e090 msgr2=0x7f393c0d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c15e090 0x7f393c0d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 >> v2:172.21.15.123:3300/0 conn(0x7f393c0d24e0 msgr2=0x7f393c071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 --2- 172.21.15.89:0/3476388355 >> v2:172.21.15.123:3300/0 conn(0x7f393c0d24e0 0x7f393c071610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c0d3020 con 0x7f393c071b50 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f39340019f0 con 0x7f393c071b50 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f393a7fc700 1 --2- 172.21.15.89:0/3476388355 >> v2:172.21.15.123:3300/0 conn(0x7f393c0d24e0 0x7f393c071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.928+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 >> v1:172.21.15.89:6789/0 conn(0x7f393c071b50 legacy=0x7f393c0b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c15e090 0x7f3924003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3924003f80 0x7f3924004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924004d90 0x7f3924005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0d2a60 con 0x7f3924003f80 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0d2d40 con 0x7f3924004d90 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392400ba40 con 0x7f393c15e090 2023-11-24T06:20:03.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393a7fc700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3924003f80 0x7f3924004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393affd700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c15e090 0x7f3924003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393b7fe700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924004d90 0x7f3924005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393a7fc700 1 -- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c15e090 msgr2=0x7f3924003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393a7fc700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c15e090 0x7f3924003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393a7fc700 1 -- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924004d90 msgr2=0x7f3924005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393a7fc700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924004d90 0x7f3924005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393a7fc700 1 -- 172.21.15.89:0/3476388355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39240035b0 con 0x7f3924003f80 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f393a7fc700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3924003f80 0x7f3924004850 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f392c009d30 tx=0x7f3924005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bc0a86dfbf81a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c009e80 con 0x7f3924003f80 2023-11-24T06:20:03.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c00e9a0 con 0x7f3924003f80 2023-11-24T06:20:03.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.929+0000 7f3939ffb700 1 -- 172.21.15.89:0/3476388355 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c00ed30 con 0x7f3924003f80 2023-11-24T06:20:03.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.930+0000 7f3941387700 1 -- 172.21.15.89:0/3476388355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3924003f80 msgr2=0x7f3924004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.930+0000 7f3941387700 1 --2- 172.21.15.89:0/3476388355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3924003f80 0x7f3924004850 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f392c009d30 tx=0x7f3924005880 comp rx=0 tx=0).stop 2023-11-24T06:20:03.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.930+0000 7f393b7fe700 1 -- 172.21.15.89:0/3476388355 reap_dead start 2023-11-24T06:20:03.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.930+0000 7f3941387700 1 -- 172.21.15.89:0/3476388355 shutdown_connections 2023-11-24T06:20:03.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.930+0000 7f3941387700 1 -- 172.21.15.89:0/3476388355 >> 172.21.15.89:0/3476388355 conn(0x7f393c06b410 msgr2=0x7f393c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:03.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.930+0000 7f3941387700 1 -- 172.21.15.89:0/3476388355 shutdown_connections 2023-11-24T06:20:03.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.930+0000 7f3941387700 1 -- 172.21.15.89:0/3476388355 wait complete. 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.931+0000 7f3941387700 1 Processor -- start 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.931+0000 7f3941387700 1 -- start start 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f3941387700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924003f80 0x7f393c0ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f3941387700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f393c1511c0 0x7f393c0b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f3941387700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c151640 0x7f393c0b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f3941387700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0cdce0 con 0x7f393c1511c0 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f3941387700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0cd0a0 con 0x7f3924003f80 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f3941387700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1f7e90 con 0x7f393c151640 2023-11-24T06:20:03.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f393a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f393c1511c0 0x7f393c0b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f393a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f393c1511c0 0x7f393c0b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37286/0 (socket says 172.21.15.89:37286) 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.932+0000 7f393a7fc700 1 -- 172.21.15.89:0/4153483490 learned_addr learned my addr 172.21.15.89:0/4153483490 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393affd700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924003f80 0x7f393c0ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393b7fe700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c151640 0x7f393c0b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393a7fc700 1 -- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c151640 msgr2=0x7f393c0b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393a7fc700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c151640 0x7f393c0b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393a7fc700 1 -- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924003f80 msgr2=0x7f393c0ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393a7fc700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924003f80 0x7f393c0ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393a7fc700 1 -- 172.21.15.89:0/4153483490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c1f8170 con 0x7f393c1511c0 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393affd700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924003f80 0x7f393c0ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393b7fe700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c151640 0x7f393c0b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f393a7fc700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f393c1511c0 0x7f393c0b47b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3924004780 tx=0x7f392c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f39237fe700 1 -- 172.21.15.89:0/4153483490 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c006e60 con 0x7f393c1511c0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f39237fe700 1 -- 172.21.15.89:0/4153483490 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c003b10 con 0x7f393c1511c0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c0cd620 con 0x7f393c1511c0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f39237fe700 1 -- 172.21.15.89:0/4153483490 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c009e80 con 0x7f393c1511c0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.933+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c1f72b0 con 0x7f393c1511c0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.934+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c068670 con 0x7f393c1511c0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.935+0000 7f39237fe700 1 -- 172.21.15.89:0/4153483490 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f392c013470 con 0x7f393c1511c0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.935+0000 7f39237fe700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f392806aed0 0x7f392806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.936+0000 7f39237fe700 1 -- 172.21.15.89:0/4153483490 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f392c08b370 con 0x7f393c1511c0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.936+0000 7f393affd700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f392806aed0 0x7f392806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.936+0000 7f393affd700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f392806aed0 0x7f392806d390 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f393c1f7210 tx=0x7f3930008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:03.939+0000 7f39237fe700 1 -- 172.21.15.89:0/4153483490 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f392c056650 con 0x7f393c1511c0 2023-11-24T06:20:04.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.131+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f393c068fb0 con 0x7f393c1511c0 2023-11-24T06:20:04.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.131+0000 7f39237fe700 1 -- 172.21.15.89:0/4153483490 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7f392c059c70 con 0x7f393c1511c0 2023-11-24T06:20:04.134 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:20:04.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f392806aed0 msgr2=0x7f392806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f392806aed0 0x7f392806d390 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f393c1f7210 tx=0x7f3930008040 comp rx=0 tx=0).stop 2023-11-24T06:20:04.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f393c1511c0 msgr2=0x7f393c0b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f393c1511c0 0x7f393c0b47b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3924004780 tx=0x7f392c0047c0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 shutdown_connections 2023-11-24T06:20:04.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f393c151640 0x7f393c0b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f392806aed0 0x7f392806d390 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3924003f80 0x7f393c0ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 --2- 172.21.15.89:0/4153483490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f393c1511c0 0x7f393c0b47b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 >> 172.21.15.89:0/4153483490 conn(0x7f393c06b410 msgr2=0x7f393c15d480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:04.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 shutdown_connections 2023-11-24T06:20:04.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.133+0000 7f3941387700 1 -- 172.21.15.89:0/4153483490 wait complete. 2023-11-24T06:20:04.148 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:20:04.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.267+0000 7f4139bcc700 1 Processor -- start 2023-11-24T06:20:04.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.267+0000 7f4139bcc700 1 -- start start 2023-11-24T06:20:04.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f4139bcc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f413415e2a0 0x7f413406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f4139bcc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413406e7f0 con 0x7f4134073f90 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f4139bcc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413406dec0 con 0x7f41340ce090 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f4139bcc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413406e1a0 con 0x7f413415e2a0 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f41337fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f413415e2a0 0x7f413406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f4133fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4134073f90 0x7f41340cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40800/0 (socket says 172.21.15.89:40800) 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f4133fff700 1 -- 172.21.15.89:0/1720365457 learned_addr learned my addr 172.21.15.89:0/1720365457 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f41337fe700 1 -- 172.21.15.89:0/1720365457 >> v1:172.21.15.123:6789/0 conn(0x7f41340ce090 legacy=0x7f413406ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f41337fe700 1 -- 172.21.15.89:0/1720365457 >> v1:172.21.15.89:6789/0 conn(0x7f4134073f90 legacy=0x7f41340cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f41337fe700 1 -- 172.21.15.89:0/1720365457 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413406e480 con 0x7f413415e2a0 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f41337fe700 1 --2- 172.21.15.89:0/1720365457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f413406ba70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4124009840 tx=0x7f4124009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347d542bce4d91dc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.268+0000 7f4132ffd700 1 -- 172.21.15.89:0/1720365457 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124004230 con 0x7f413415e2a0 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4132ffd700 1 -- 172.21.15.89:0/1720365457 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4124004390 con 0x7f413415e2a0 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4132ffd700 1 -- 172.21.15.89:0/1720365457 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41240036a0 con 0x7f413415e2a0 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4139bcc700 1 -- 172.21.15.89:0/1720365457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 msgr2=0x7f413406ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4139bcc700 1 --2- 172.21.15.89:0/1720365457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f413406ba70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4124009840 tx=0x7f4124009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4139bcc700 1 -- 172.21.15.89:0/1720365457 shutdown_connections 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4139bcc700 1 --2- 172.21.15.89:0/1720365457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f413406ba70 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4139bcc700 1 -- 172.21.15.89:0/1720365457 >> 172.21.15.89:0/1720365457 conn(0x7f41340cd570 msgr2=0x7f41340cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4139bcc700 1 -- 172.21.15.89:0/1720365457 shutdown_connections 2023-11-24T06:20:04.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.269+0000 7f4139bcc700 1 -- 172.21.15.89:0/1720365457 wait complete. 2023-11-24T06:20:04.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.270+0000 7f4139bcc700 1 Processor -- start 2023-11-24T06:20:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.270+0000 7f4139bcc700 1 -- start start 2023-11-24T06:20:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4139bcc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4134073f90 0x7f4134158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4139bcc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41340ce090 0x7f4134148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4139bcc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f41340ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4139bcc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4134157a70 con 0x7f413415e2a0 2023-11-24T06:20:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4139bcc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4134145ea0 con 0x7f41340ce090 2023-11-24T06:20:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4139bcc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41341490f0 con 0x7f4134073f90 2023-11-24T06:20:04.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4133fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f41340ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4133fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f41340ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37302/0 (socket says 172.21.15.89:37302) 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f4133fff700 1 -- 172.21.15.89:0/1827972593 learned_addr learned my addr 172.21.15.89:0/1827972593 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f412affd700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41340ce090 0x7f4134148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.271+0000 7f41337fe700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4134073f90 0x7f4134158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4133fff700 1 -- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4134073f90 msgr2=0x7f4134158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4133fff700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4134073f90 0x7f4134158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4133fff700 1 -- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41340ce090 msgr2=0x7f4134148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4133fff700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41340ce090 0x7f4134148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4133fff700 1 -- 172.21.15.89:0/1827972593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240094a0 con 0x7f413415e2a0 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f412affd700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41340ce090 0x7f4134148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f41337fe700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4134073f90 0x7f4134158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:04.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4133fff700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f41340ae9f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f412c00cbc0 tx=0x7f412c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4130ff9700 1 -- 172.21.15.89:0/1827972593 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c00ab50 con 0x7f413415e2a0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4130ff9700 1 -- 172.21.15.89:0/1827972593 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c00acb0 con 0x7f413415e2a0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4134157ca0 con 0x7f413415e2a0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4130ff9700 1 -- 172.21.15.89:0/1827972593 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c012600 con 0x7f413415e2a0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.272+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4134158260 con 0x7f413415e2a0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.273+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4134159280 con 0x7f413415e2a0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.273+0000 7f4130ff9700 1 -- 172.21.15.89:0/1827972593 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f412c01b3b0 con 0x7f413415e2a0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.274+0000 7f4130ff9700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f411406aed0 0x7f411406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.274+0000 7f4130ff9700 1 -- 172.21.15.89:0/1827972593 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f412c089270 con 0x7f413415e2a0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.274+0000 7f41337fe700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f411406aed0 0x7f411406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.274+0000 7f41337fe700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f411406aed0 0x7f411406d390 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f41240090b0 tx=0x7f4124013040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:04.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.278+0000 7f4130ff9700 1 -- 172.21.15.89:0/1827972593 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f412c054370 con 0x7f413415e2a0 2023-11-24T06:20:04.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.482+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4134067f60 con 0x7f413415e2a0 2023-11-24T06:20:04.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.483+0000 7f4130ff9700 1 -- 172.21.15.89:0/1827972593 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) v1 ==== 80+0+122 (secure 0 0 0) 0x7f412c057990 con 0x7f413415e2a0 2023-11-24T06:20:04.486 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6837/2053513777 2023-11-25T06:19:59.514768+0000 2023-11-24T06:20:04.487 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6835/2053513777 2023-11-25T06:19:59.514768+0000 2023-11-24T06:20:04.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f411406aed0 msgr2=0x7f411406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f411406aed0 0x7f411406d390 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f41240090b0 tx=0x7f4124013040 comp rx=0 tx=0).stop 2023-11-24T06:20:04.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 msgr2=0x7f41340ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f41340ae9f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f412c00cbc0 tx=0x7f412c00cf80 comp rx=0 tx=0).stop 2023-11-24T06:20:04.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 shutdown_connections 2023-11-24T06:20:04.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f413415e2a0 0x7f41340ae9f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4134073f90 0x7f4134158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f411406aed0 0x7f411406d390 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 --2- 172.21.15.89:0/1827972593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41340ce090 0x7f4134148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 >> 172.21.15.89:0/1827972593 conn(0x7f41340cd570 msgr2=0x7f4134159490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:04.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 shutdown_connections 2023-11-24T06:20:04.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.485+0000 7f4139bcc700 1 -- 172.21.15.89:0/1827972593 wait complete. 2023-11-24T06:20:04.491 INFO:teuthology.orchestra.run.smithi089.stderr:listed 2 entries 2023-11-24T06:20:04.500 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T06:20:04.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:04.501 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T06:20:04.541 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:20:04.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:04.541 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T06:20:04.599 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:04.599 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:04.621 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:04.621 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:04.681 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:04.802 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.799+0000 7fe6ef529700 1 Processor -- start 2023-11-24T06:20:04.802 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.799+0000 7fe6ef529700 1 -- start start 2023-11-24T06:20:04.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6ef529700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e815e080 0x7fe6e80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6ef529700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe6e806c710 0x7fe6e80afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6ef529700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80ae430 con 0x7fe6e80af4d0 2023-11-24T06:20:04.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6ef529700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80af0b0 con 0x7fe6e806c710 2023-11-24T06:20:04.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6ef529700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80b05a0 con 0x7fe6e815e080 2023-11-24T06:20:04.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6edac6700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe6e806c710 0x7fe6e80afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6ed2c5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e815e080 0x7fe6e80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6ecac4700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe6e80af4d0 0x7fe6e806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:44634/0 (socket says 172.21.15.89:44634) 2023-11-24T06:20:04.804 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6ecac4700 1 -- 172.21.15.89:0/1072232153 learned_addr learned my addr 172.21.15.89:0/1072232153 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:04.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6edac6700 1 -- 172.21.15.89:0/1072232153 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e815e080 msgr2=0x7fe6e80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:04.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6edac6700 1 --2- 172.21.15.89:0/1072232153 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e815e080 0x7fe6e80b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.800+0000 7fe6edac6700 1 -- 172.21.15.89:0/1072232153 >> v1:172.21.15.123:6789/0 conn(0x7fe6e80af4d0 legacy=0x7fe6e806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6edac6700 1 -- 172.21.15.89:0/1072232153 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80ae710 con 0x7fe6e806c710 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6dffff700 1 -- 172.21.15.89:0/1072232153 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753949385 0 0) 0x7fe6e80ae430 con 0x7fe6e80af4d0 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6edac6700 1 --2- 172.21.15.89:0/1072232153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e806c710 0x7fe6e80afb90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009dc0 tx=0x7fe6e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c372a8802b79a8f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6dffff700 1 -- 172.21.15.89:0/1072232153 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e000e070 con 0x7fe6e806c710 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6dffff700 1 -- 172.21.15.89:0/1072232153 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e000bae0 con 0x7fe6e806c710 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6dffff700 1 -- 172.21.15.89:0/1072232153 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e000be90 con 0x7fe6e806c710 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6ef529700 1 -- 172.21.15.89:0/1072232153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e806c710 msgr2=0x7fe6e80afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6ef529700 1 --2- 172.21.15.89:0/1072232153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e806c710 0x7fe6e80afb90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009dc0 tx=0x7fe6e000b330 comp rx=0 tx=0).stop 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6ef529700 1 -- 172.21.15.89:0/1072232153 shutdown_connections 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6ef529700 1 --2- 172.21.15.89:0/1072232153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e806c710 0x7fe6e80afb90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6ef529700 1 --2- 172.21.15.89:0/1072232153 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e815e080 0x7fe6e80b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.801+0000 7fe6ef529700 1 -- 172.21.15.89:0/1072232153 >> 172.21.15.89:0/1072232153 conn(0x7fe6e8071b90 msgr2=0x7fe6e8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:04.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.802+0000 7fe6ef529700 1 -- 172.21.15.89:0/1072232153 shutdown_connections 2023-11-24T06:20:04.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.802+0000 7fe6ef529700 1 -- 172.21.15.89:0/1072232153 wait complete. 2023-11-24T06:20:04.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.803+0000 7fe6ef529700 1 Processor -- start 2023-11-24T06:20:04.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.803+0000 7fe6ef529700 1 -- start start 2023-11-24T06:20:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.803+0000 7fe6ef529700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e806c710 0x7fe6e8150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.803+0000 7fe6ef529700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e80af4d0 0x7fe6e81545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.803+0000 7fe6ef529700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e815e080 0x7fe6e80c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.803+0000 7fe6ef529700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81f7a90 con 0x7fe6e815e080 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.803+0000 7fe6ef529700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8154b10 con 0x7fe6e80af4d0 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.803+0000 7fe6ef529700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81f6f10 con 0x7fe6e806c710 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e815e080 0x7fe6e80c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e815e080 0x7fe6e80c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37316/0 (socket says 172.21.15.89:37316) 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 -- 172.21.15.89:0/4035950578 learned_addr learned my addr 172.21.15.89:0/4035950578 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6ed2c5700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e806c710 0x7fe6e8150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6ecac4700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e80af4d0 0x7fe6e81545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 -- 172.21.15.89:0/4035950578 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e806c710 msgr2=0x7fe6e8150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e806c710 0x7fe6e8150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 -- 172.21.15.89:0/4035950578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e80af4d0 msgr2=0x7fe6e81545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:04.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e80af4d0 0x7fe6e81545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 -- 172.21.15.89:0/4035950578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e0009a70 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6edac6700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e815e080 0x7fe6e80c6f40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0014040 tx=0x7fe6e0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6ddffb700 1 -- 172.21.15.89:0/4035950578 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0004610 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6ddffb700 1 -- 172.21.15.89:0/4035950578 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e0020070 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6ddffb700 1 -- 172.21.15.89:0/4035950578 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0027480 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e81f71f0 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.804+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e81f77b0 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.805+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e80753f0 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.806+0000 7fe6ddffb700 1 -- 172.21.15.89:0/4035950578 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe6e0012430 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.806+0000 7fe6ddffb700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6cc06aed0 0x7fe6cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.806+0000 7fe6ddffb700 1 -- 172.21.15.89:0/4035950578 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe6e0022070 con 0x7fe6e815e080 2023-11-24T06:20:04.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.806+0000 7fe6ed2c5700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6cc06aed0 0x7fe6cc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:04.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.806+0000 7fe6ed2c5700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6cc06aed0 0x7fe6cc06d390 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009ad0 tx=0x7fe6e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:04.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:04.810+0000 7fe6ddffb700 1 -- 172.21.15.89:0/4035950578 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6e0056440 con 0x7fe6e815e080 2023-11-24T06:20:05.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.003+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6e8067f60 con 0x7fe6e815e080 2023-11-24T06:20:05.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.004+0000 7fe6ddffb700 1 -- 172.21.15.89:0/4035950578 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16662 (secure 0 0 0) 0x7fe6e0059a60 con 0x7fe6e815e080 2023-11-24T06:20:05.007 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:05.007 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:02.530677+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/2053513777":"2023-11-25T06:19:59.514768+0000","172.21.15.89:6835/2053513777":"2023-11-25T06:19:59.514768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:05.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6cc06aed0 msgr2=0x7fe6cc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6cc06aed0 0x7fe6cc06d390 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009ad0 tx=0x7fe6e4008040 comp rx=0 tx=0).stop 2023-11-24T06:20:05.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e815e080 msgr2=0x7fe6e80c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e815e080 0x7fe6e80c6f40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0014040 tx=0x7fe6e0004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 shutdown_connections 2023-11-24T06:20:05.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e815e080 0x7fe6e80c6f40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6e806c710 0x7fe6e8150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6cc06aed0 0x7fe6cc06d390 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 --2- 172.21.15.89:0/4035950578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e80af4d0 0x7fe6e81545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 >> 172.21.15.89:0/4035950578 conn(0x7fe6e8071b90 msgr2=0x7fe6e806e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:05.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 shutdown_connections 2023-11-24T06:20:05.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.006+0000 7fe6ef529700 1 -- 172.21.15.89:0/4035950578 wait complete. 2023-11-24T06:20:05.021 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:19:41.783017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:19:42.953803+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:05.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.142+0000 7f36c7ba8700 1 Processor -- start 2023-11-24T06:20:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.142+0000 7f36c7ba8700 1 -- start start 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c7ba8700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f36c00cf610 0x7f36c00add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c7ba8700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f36c015f9e0 0x7f36c00b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c7ba8700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c006b080 con 0x7f36c0073f90 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c7ba8700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c015c290 con 0x7f36c015f9e0 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c7ba8700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c00cf9f0 con 0x7f36c00cf610 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c5143700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f36c015f9e0 0x7f36c00b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c5143700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f36c015f9e0 0x7f36c00b3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37324/0 (socket says 172.21.15.89:37324) 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c5143700 1 -- 172.21.15.89:0/95043800 learned_addr learned my addr 172.21.15.89:0/95043800 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.143+0000 7f36c5944700 1 --2- 172.21.15.89:0/95043800 >> v2:172.21.15.123:3300/0 conn(0x7f36c00cf610 0x7f36c00add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c5143700 1 -- 172.21.15.89:0/95043800 >> v2:172.21.15.123:3300/0 conn(0x7f36c00cf610 msgr2=0x7f36c00add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c5143700 1 --2- 172.21.15.89:0/95043800 >> v2:172.21.15.123:3300/0 conn(0x7f36c00cf610 0x7f36c00add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c5143700 1 -- 172.21.15.89:0/95043800 >> v1:172.21.15.123:6789/0 conn(0x7f36c0073f90 legacy=0x7f36c006b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c5143700 1 -- 172.21.15.89:0/95043800 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c015ff80 con 0x7f36c015f9e0 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c5944700 1 --2- 172.21.15.89:0/95043800 >> v2:172.21.15.123:3300/0 conn(0x7f36c00cf610 0x7f36c00add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c5143700 1 --2- 172.21.15.89:0/95043800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c015f9e0 0x7f36c00b3c60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f36b00098f0 tx=0x7f36b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afee3943422cd9aa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c4942700 1 -- 172.21.15.89:0/95043800 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b000b860 con 0x7f36c015f9e0 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c4942700 1 -- 172.21.15.89:0/95043800 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b000b9c0 con 0x7f36c015f9e0 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c4942700 1 -- 172.21.15.89:0/95043800 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b000bd50 con 0x7f36c015f9e0 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c7ba8700 1 -- 172.21.15.89:0/95043800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c015f9e0 msgr2=0x7f36c00b3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/95043800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c015f9e0 0x7f36c00b3c60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f36b00098f0 tx=0x7f36b0009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c7ba8700 1 -- 172.21.15.89:0/95043800 shutdown_connections 2023-11-24T06:20:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/95043800 >> v2:172.21.15.123:3300/0 conn(0x7f36c00cf610 0x7f36c00add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/95043800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c015f9e0 0x7f36c00b3c60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.144+0000 7f36c7ba8700 1 -- 172.21.15.89:0/95043800 >> 172.21.15.89:0/95043800 conn(0x7f36c00d0bf0 msgr2=0x7f36c00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.145+0000 7f36c7ba8700 1 -- 172.21.15.89:0/95043800 shutdown_connections 2023-11-24T06:20:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.145+0000 7f36c7ba8700 1 -- 172.21.15.89:0/95043800 wait complete. 2023-11-24T06:20:05.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.146+0000 7f36c7ba8700 1 Processor -- start 2023-11-24T06:20:05.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.146+0000 7f36c7ba8700 1 -- start start 2023-11-24T06:20:05.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c7ba8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36c0073f90 0x7f36c0159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c7ba8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c00cf610 0x7f36c00b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c7ba8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36c015f9e0 0x7f36c00b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c7ba8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c009e850 con 0x7f36c00cf610 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c7ba8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c00c6070 con 0x7f36c0073f90 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c7ba8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c009e290 con 0x7f36c015f9e0 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c00cf610 0x7f36c00b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c00cf610 0x7f36c00b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37328/0 (socket says 172.21.15.89:37328) 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 -- 172.21.15.89:0/2837239075 learned_addr learned my addr 172.21.15.89:0/2837239075 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5944700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36c0073f90 0x7f36c0159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c6145700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36c015f9e0 0x7f36c00b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 -- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36c015f9e0 msgr2=0x7f36c00b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36c015f9e0 0x7f36c00b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 -- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36c0073f90 msgr2=0x7f36c0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36c0073f90 0x7f36c0159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 -- 172.21.15.89:0/2837239075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0009550 con 0x7f36c00cf610 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c6145700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36c015f9e0 0x7f36c00b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5944700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36c0073f90 0x7f36c0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.147+0000 7f36c5143700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c00cf610 0x7f36c00b4410 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f36b0000c00 tx=0x7f36b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.148+0000 7f36b67fc700 1 -- 172.21.15.89:0/2837239075 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b0004340 con 0x7f36c00cf610 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.148+0000 7f36b67fc700 1 -- 172.21.15.89:0/2837239075 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b001f070 con 0x7f36c00cf610 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.148+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c009e570 con 0x7f36c00cf610 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.148+0000 7f36b67fc700 1 -- 172.21.15.89:0/2837239075 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b00036a0 con 0x7f36c00cf610 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.148+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c01595a0 con 0x7f36c00cf610 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.149+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c00b0670 con 0x7f36c00cf610 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.149+0000 7f36b67fc700 1 -- 172.21.15.89:0/2837239075 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f36b0021030 con 0x7f36c00cf610 2023-11-24T06:20:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.149+0000 7f36b67fc700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f36a406aed0 0x7f36a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.150+0000 7f36b67fc700 1 -- 172.21.15.89:0/2837239075 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f36b008aa10 con 0x7f36c00cf610 2023-11-24T06:20:05.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.150+0000 7f36c5944700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f36a406aed0 0x7f36a406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.150+0000 7f36c5944700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f36a406aed0 0x7f36a406d390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f36c01f5410 tx=0x7f36bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.154+0000 7f36b67fc700 1 -- 172.21.15.89:0/2837239075 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36b0055a70 con 0x7f36c00cf610 2023-11-24T06:20:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.345+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f36c00c61e0 con 0x7f36c00cf610 2023-11-24T06:20:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.345+0000 7f36b67fc700 1 -- 172.21.15.89:0/2837239075 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f36b0059090 con 0x7f36c00cf610 2023-11-24T06:20:05.349 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:05.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.347+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f36a406aed0 msgr2=0x7f36a406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.347+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f36a406aed0 0x7f36a406d390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f36c01f5410 tx=0x7f36bc008040 comp rx=0 tx=0).stop 2023-11-24T06:20:05.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c00cf610 msgr2=0x7f36c00b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c00cf610 0x7f36c00b4410 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f36b0000c00 tx=0x7f36b0004750 comp rx=0 tx=0).stop 2023-11-24T06:20:05.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 shutdown_connections 2023-11-24T06:20:05.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36c015f9e0 0x7f36c00b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f36a406aed0 0x7f36a406d390 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36c0073f90 0x7f36c0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 --2- 172.21.15.89:0/2837239075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c00cf610 0x7f36c00b4410 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 >> 172.21.15.89:0/2837239075 conn(0x7f36c00d0bf0 msgr2=0x7f36c00aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:05.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 shutdown_connections 2023-11-24T06:20:05.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.348+0000 7f36c7ba8700 1 -- 172.21.15.89:0/2837239075 wait complete. 2023-11-24T06:20:05.363 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:20:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.483+0000 7f590be7d700 1 Processor -- start 2023-11-24T06:20:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.483+0000 7f590be7d700 1 -- start start 2023-11-24T06:20:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f590be7d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f59040b49a0 0x7f590415b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f590be7d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f590406e370 0x7f59040b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f590be7d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e880 0x7f590406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f590be7d700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590415e9f0 con 0x7f590406e370 2023-11-24T06:20:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f590be7d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59040b52c0 con 0x7f59040b49a0 2023-11-24T06:20:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f590be7d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590415ec90 con 0x7f590406e880 2023-11-24T06:20:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f590406e370 0x7f59040b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f590406e370 0x7f59040b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37334/0 (socket says 172.21.15.89:37334) 2023-11-24T06:20:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 -- 172.21.15.89:0/418621107 learned_addr learned my addr 172.21.15.89:0/418621107 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 -- 172.21.15.89:0/418621107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e880 msgr2=0x7f590406ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909c19700 1 --2- 172.21.15.89:0/418621107 >> v2:172.21.15.123:3300/0 conn(0x7f59040b49a0 0x7f590415b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f590a41a700 1 --2- 172.21.15.89:0/418621107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e880 0x7f590406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 --2- 172.21.15.89:0/418621107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e880 0x7f590406ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 -- 172.21.15.89:0/418621107 >> v2:172.21.15.123:3300/0 conn(0x7f59040b49a0 msgr2=0x7f590415b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 --2- 172.21.15.89:0/418621107 >> v2:172.21.15.123:3300/0 conn(0x7f59040b49a0 0x7f590415b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 -- 172.21.15.89:0/418621107 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590415ef70 con 0x7f590406e370 2023-11-24T06:20:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.484+0000 7f5909418700 1 --2- 172.21.15.89:0/418621107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590406e370 0x7f59040b4d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5904072f40 tx=0x7f590000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451dbcc1b230e1ed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f5908c17700 1 -- 172.21.15.89:0/418621107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590000dd30 con 0x7f590406e370 2023-11-24T06:20:05.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f5908c17700 1 -- 172.21.15.89:0/418621107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590000de90 con 0x7f590406e370 2023-11-24T06:20:05.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f5908c17700 1 -- 172.21.15.89:0/418621107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900010690 con 0x7f590406e370 2023-11-24T06:20:05.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 -- 172.21.15.89:0/418621107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590406e370 msgr2=0x7f59040b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 --2- 172.21.15.89:0/418621107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590406e370 0x7f59040b4d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5904072f40 tx=0x7f590000be70 comp rx=0 tx=0).stop 2023-11-24T06:20:05.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 -- 172.21.15.89:0/418621107 shutdown_connections 2023-11-24T06:20:05.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 --2- 172.21.15.89:0/418621107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e880 0x7f590406ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 --2- 172.21.15.89:0/418621107 >> v2:172.21.15.123:3300/0 conn(0x7f59040b49a0 0x7f590415b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 --2- 172.21.15.89:0/418621107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590406e370 0x7f59040b4d80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 -- 172.21.15.89:0/418621107 >> 172.21.15.89:0/418621107 conn(0x7f59040b28f0 msgr2=0x7f59040b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:05.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 -- 172.21.15.89:0/418621107 shutdown_connections 2023-11-24T06:20:05.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.485+0000 7f590be7d700 1 -- 172.21.15.89:0/418621107 wait complete. 2023-11-24T06:20:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.486+0000 7f590be7d700 1 Processor -- start 2023-11-24T06:20:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590be7d700 1 -- start start 2023-11-24T06:20:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590be7d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e370 0x7f5904071e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590be7d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f590406e880 0x7f5904158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590be7d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040b49a0 0x7f5904143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590be7d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040ae4a0 con 0x7f59040b49a0 2023-11-24T06:20:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590be7d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040ae780 con 0x7f590406e880 2023-11-24T06:20:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590be7d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040aea60 con 0x7f590406e370 2023-11-24T06:20:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590a41a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040b49a0 0x7f5904143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590a41a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040b49a0 0x7f5904143130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37344/0 (socket says 172.21.15.89:37344) 2023-11-24T06:20:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f590a41a700 1 -- 172.21.15.89:0/1820319183 learned_addr learned my addr 172.21.15.89:0/1820319183 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f5909418700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f590406e880 0x7f5904158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.487+0000 7f5909c19700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e370 0x7f5904071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f590a41a700 1 -- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e370 msgr2=0x7f5904071e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f590a41a700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e370 0x7f5904071e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f590a41a700 1 -- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f590406e880 msgr2=0x7f5904158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f590a41a700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f590406e880 0x7f5904158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f590a41a700 1 -- 172.21.15.89:0/1820319183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590000b7f0 con 0x7f59040b49a0 2023-11-24T06:20:05.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f5909c19700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e370 0x7f5904071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:05.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f590a41a700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040b49a0 0x7f5904143130 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00cb80 tx=0x7f58fc00ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f58fa7fc700 1 -- 172.21.15.89:0/1820319183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc00a820 con 0x7f59040b49a0 2023-11-24T06:20:05.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f58fa7fc700 1 -- 172.21.15.89:0/1820319183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc00a980 con 0x7f59040b49a0 2023-11-24T06:20:05.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59040aed40 con 0x7f59040b49a0 2023-11-24T06:20:05.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f58fa7fc700 1 -- 172.21.15.89:0/1820319183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc00ad30 con 0x7f59040b49a0 2023-11-24T06:20:05.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.488+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590414e1e0 con 0x7f59040b49a0 2023-11-24T06:20:05.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.489+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590414e3d0 con 0x7f59040b49a0 2023-11-24T06:20:05.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.489+0000 7f58fa7fc700 1 -- 172.21.15.89:0/1820319183 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f58fc011410 con 0x7f59040b49a0 2023-11-24T06:20:05.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.490+0000 7f58fa7fc700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f58e806aed0 0x7f58e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.490+0000 7f58fa7fc700 1 -- 172.21.15.89:0/1820319183 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f58fc0891d0 con 0x7f59040b49a0 2023-11-24T06:20:05.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.490+0000 7f5909c19700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f58e806aed0 0x7f58e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.490+0000 7f5909c19700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f58e806aed0 0x7f58e806d390 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f58f400aa50 tx=0x7f58f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.494+0000 7f58fa7fc700 1 -- 172.21.15.89:0/1820319183 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58fc054220 con 0x7f59040b49a0 2023-11-24T06:20:05.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.683+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5904068fb0 con 0x7f59040b49a0 2023-11-24T06:20:05.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.684+0000 7f58fa7fc700 1 -- 172.21.15.89:0/1820319183 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7f58fc057840 con 0x7f59040b49a0 2023-11-24T06:20:05.687 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:20:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f58e806aed0 msgr2=0x7f58e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f58e806aed0 0x7f58e806d390 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f58f400aa50 tx=0x7f58f4008040 comp rx=0 tx=0).stop 2023-11-24T06:20:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040b49a0 msgr2=0x7f5904143130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040b49a0 0x7f5904143130 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00cb80 tx=0x7f58fc00ce90 comp rx=0 tx=0).stop 2023-11-24T06:20:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 shutdown_connections 2023-11-24T06:20:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040b49a0 0x7f5904143130 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f590406e370 0x7f5904071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f58e806aed0 0x7f58e806d390 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 --2- 172.21.15.89:0/1820319183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f590406e880 0x7f5904158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 >> 172.21.15.89:0/1820319183 conn(0x7f59040b28f0 msgr2=0x7f59040748a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 shutdown_connections 2023-11-24T06:20:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.686+0000 7f590be7d700 1 -- 172.21.15.89:0/1820319183 wait complete. 2023-11-24T06:20:05.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:20:05.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.821+0000 7f9dda0ab700 1 Processor -- start 2023-11-24T06:20:05.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.821+0000 7f9dda0ab700 1 -- start start 2023-11-24T06:20:05.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dda0ab700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40b4370 0x7f9dd415b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dda0ab700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd40ae5e0 con 0x7f9dd41587f0 2023-11-24T06:20:05.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dda0ab700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd415be30 con 0x7f9dd415b370 2023-11-24T06:20:05.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dda0ab700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd415c110 con 0x7f9dd40b4370 2023-11-24T06:20:05.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd37fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9dd41587f0 0x7f9dd406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40852/0 (socket says 172.21.15.89:40852) 2023-11-24T06:20:05.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd37fe700 1 -- 172.21.15.89:0/2100746361 learned_addr learned my addr 172.21.15.89:0/2100746361 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd3fff700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40b4370 0x7f9dd415b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229289289 0 0) 0x7f9dd40ae5e0 con 0x7f9dd41587f0 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dbc0035b0 con 0x7f9dd41587f0 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705263243 0 0) 0x7f9dbc0035b0 con 0x7f9dd41587f0 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40b4370 msgr2=0x7f9dd415b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd27fc700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40b4370 0x7f9dd415b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 >> v1:172.21.15.123:6789/0 conn(0x7f9dd415b370 legacy=0x7f9dd40b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.822+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40ae860 con 0x7f9dd41587f0 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f9dc8001670 con 0x7f9dd41587f0 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 >> v1:172.21.15.89:6789/0 conn(0x7f9dd41587f0 legacy=0x7f9dd406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40b4370 0x7f9dbc003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dbc003f70 0x7f9dbc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dbc004d90 0x7f9dbc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd415c110 con 0x7f9dbc004d90 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc00ba70 con 0x7f9dbc003f70 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc00bc30 con 0x7f9dd40b4370 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd3fff700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dbc004d90 0x7f9dbc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd37fe700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40b4370 0x7f9dbc003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd2ffd700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dbc003f70 0x7f9dbc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd3fff700 1 -- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40b4370 msgr2=0x7f9dbc003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd3fff700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40b4370 0x7f9dbc003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd3fff700 1 -- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dbc003f70 msgr2=0x7f9dbc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd3fff700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dbc003f70 0x7f9dbc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd3fff700 1 -- 172.21.15.89:0/2100746361 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc0035b0 con 0x7f9dbc004d90 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd3fff700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dbc004d90 0x7f9dbc005210 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc009f90 tx=0x7f9dbc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb8e40508217ec1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc00ec30 con 0x7f9dbc004d90 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc00f040 con 0x7f9dbc004d90 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.823+0000 7f9dd27fc700 1 -- 172.21.15.89:0/2100746361 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc014480 con 0x7f9dbc004d90 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.824+0000 7f9dda0ab700 1 -- 172.21.15.89:0/2100746361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dbc004d90 msgr2=0x7f9dbc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.824+0000 7f9dda0ab700 1 --2- 172.21.15.89:0/2100746361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dbc004d90 0x7f9dbc005210 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc009f90 tx=0x7f9dbc005880 comp rx=0 tx=0).stop 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.824+0000 7f9dd3fff700 1 -- 172.21.15.89:0/2100746361 reap_dead start 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.824+0000 7f9dda0ab700 1 -- 172.21.15.89:0/2100746361 shutdown_connections 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.824+0000 7f9dda0ab700 1 -- 172.21.15.89:0/2100746361 >> 172.21.15.89:0/2100746361 conn(0x7f9dd40d0bf0 msgr2=0x7f9dd40d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.824+0000 7f9dda0ab700 1 -- 172.21.15.89:0/2100746361 shutdown_connections 2023-11-24T06:20:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.824+0000 7f9dda0ab700 1 -- 172.21.15.89:0/2100746361 wait complete. 2023-11-24T06:20:05.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.825+0000 7f9dda0ab700 1 Processor -- start 2023-11-24T06:20:05.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.825+0000 7f9dda0ab700 1 -- start start 2023-11-24T06:20:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dda0ab700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd4074200 0x7f9dd41f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dda0ab700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40745e0 0x7f9dd415dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dda0ab700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dd41482e0 0x7f9dd4148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dda0ab700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd415e290 con 0x7f9dd41482e0 2023-11-24T06:20:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dda0ab700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41f8210 con 0x7f9dd4074200 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dda0ab700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41f7970 con 0x7f9dd40745e0 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dd41482e0 0x7f9dd4148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dd41482e0 0x7f9dd4148770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37364/0 (socket says 172.21.15.89:37364) 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 -- 172.21.15.89:0/949698749 learned_addr learned my addr 172.21.15.89:0/949698749 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd37fe700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd4074200 0x7f9dd41f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd2ffd700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40745e0 0x7f9dd415dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 -- 172.21.15.89:0/949698749 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40745e0 msgr2=0x7f9dd415dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40745e0 0x7f9dd415dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 -- 172.21.15.89:0/949698749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd4074200 msgr2=0x7f9dd41f72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd4074200 0x7f9dd41f72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 -- 172.21.15.89:0/949698749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd41f7c50 con 0x7f9dd41482e0 2023-11-24T06:20:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.826+0000 7f9dd3fff700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dd41482e0 0x7f9dd4148770 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc016000 tx=0x7f9dcc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.827+0000 7f9dbbfff700 1 -- 172.21.15.89:0/949698749 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc007000 con 0x7f9dd41482e0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.827+0000 7f9dbbfff700 1 -- 172.21.15.89:0/949698749 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc003e50 con 0x7f9dd41482e0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.827+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd41f7f30 con 0x7f9dd41482e0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.827+0000 7f9dbbfff700 1 -- 172.21.15.89:0/949698749 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc015b60 con 0x7f9dd41482e0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.827+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd4148cb0 con 0x7f9dd41482e0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.828+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd4068fb0 con 0x7f9dd41482e0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.828+0000 7f9dbbfff700 1 -- 172.21.15.89:0/949698749 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9dcc024070 con 0x7f9dd41482e0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.828+0000 7f9dbbfff700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9dc006aed0 0x7f9dc006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.828+0000 7f9dbbfff700 1 -- 172.21.15.89:0/949698749 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9dcc08b150 con 0x7f9dd41482e0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.828+0000 7f9dd37fe700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9dc006aed0 0x7f9dc006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:05.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.829+0000 7f9dd37fe700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9dc006aed0 0x7f9dc006d390 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8000c00 tx=0x7f9dc8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:05.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:05.833+0000 7f9dbbfff700 1 -- 172.21.15.89:0/949698749 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9dcc056230 con 0x7f9dd41482e0 2023-11-24T06:20:06.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.022+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9dd4157770 con 0x7f9dd41482e0 2023-11-24T06:20:06.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.023+0000 7f9dbbfff700 1 -- 172.21.15.89:0/949698749 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9dcc059850 con 0x7f9dd41482e0 2023-11-24T06:20:06.026 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:20:06.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9dc006aed0 msgr2=0x7f9dc006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9dc006aed0 0x7f9dc006d390 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8000c00 tx=0x7f9dc8009040 comp rx=0 tx=0).stop 2023-11-24T06:20:06.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dd41482e0 msgr2=0x7f9dd4148770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dd41482e0 0x7f9dd4148770 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc016000 tx=0x7f9dcc009750 comp rx=0 tx=0).stop 2023-11-24T06:20:06.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 shutdown_connections 2023-11-24T06:20:06.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dd41482e0 0x7f9dd4148770 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9dc006aed0 0x7f9dc006d390 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd4074200 0x7f9dd41f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 --2- 172.21.15.89:0/949698749 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9dd40745e0 0x7f9dd415dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 >> 172.21.15.89:0/949698749 conn(0x7f9dd40d0bf0 msgr2=0x7f9dd415a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:06.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 shutdown_connections 2023-11-24T06:20:06.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.025+0000 7f9dda0ab700 1 -- 172.21.15.89:0/949698749 wait complete. 2023-11-24T06:20:06.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:20:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.160+0000 7f695456a700 1 Processor -- start 2023-11-24T06:20:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.160+0000 7f695456a700 1 -- start start 2023-11-24T06:20:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f695456a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f694c15bdd0 0x7f694c1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f695456a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0d1c70 0x7f694c0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f695456a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c0b22d0 con 0x7f694c0b3370 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f695456a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f694c0b2f50 con 0x7f694c15bdd0 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f695456a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c0d2640 con 0x7f694c0d1c70 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6952306700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f694c15bdd0 0x7f694c1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6951b05700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f694c0b3370 0x7f694c0b1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40872/0 (socket says 172.21.15.89:40872) 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6951b05700 1 -- 172.21.15.89:0/2894593573 learned_addr learned my addr 172.21.15.89:0/2894593573 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6952b07700 1 --2- 172.21.15.89:0/2894593573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0d1c70 0x7f694c0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6952306700 1 -- 172.21.15.89:0/2894593573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0d1c70 msgr2=0x7f694c0d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6952306700 1 --2- 172.21.15.89:0/2894593573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0d1c70 0x7f694c0d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6952306700 1 -- 172.21.15.89:0/2894593573 >> v1:172.21.15.89:6789/0 conn(0x7f694c0b3370 legacy=0x7f694c0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6952306700 1 -- 172.21.15.89:0/2894593573 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c0b25b0 con 0x7f694c15bdd0 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6951304700 1 -- 172.21.15.89:0/2894593573 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919041235 0 0) 0x7f694c0b22d0 con 0x7f694c0b3370 2023-11-24T06:20:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.161+0000 7f6952b07700 1 --2- 172.21.15.89:0/2894593573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0d1c70 0x7f694c0d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f6952306700 1 --2- 172.21.15.89:0/2894593573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c1604e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6948009840 tx=0x7f6948009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482f4a8090cc9b49 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f6951304700 1 -- 172.21.15.89:0/2894593573 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948011830 con 0x7f694c15bdd0 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f6951304700 1 -- 172.21.15.89:0/2894593573 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6948011990 con 0x7f694c15bdd0 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f6951304700 1 -- 172.21.15.89:0/2894593573 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948011d20 con 0x7f694c15bdd0 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f695456a700 1 -- 172.21.15.89:0/2894593573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 msgr2=0x7f694c1604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f695456a700 1 --2- 172.21.15.89:0/2894593573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c1604e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6948009840 tx=0x7f6948009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f695456a700 1 -- 172.21.15.89:0/2894593573 shutdown_connections 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f695456a700 1 --2- 172.21.15.89:0/2894593573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0d1c70 0x7f694c0d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f695456a700 1 --2- 172.21.15.89:0/2894593573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c1604e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f695456a700 1 -- 172.21.15.89:0/2894593573 >> 172.21.15.89:0/2894593573 conn(0x7f694c0731a0 msgr2=0x7f694c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f695456a700 1 -- 172.21.15.89:0/2894593573 shutdown_connections 2023-11-24T06:20:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.162+0000 7f695456a700 1 -- 172.21.15.89:0/2894593573 wait complete. 2023-11-24T06:20:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.164+0000 7f695456a700 1 Processor -- start 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.164+0000 7f695456a700 1 -- start start 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f695456a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0b3370 0x7f694c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f695456a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f694c0d1c70 0x7f694c157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f695456a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f695456a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c157780 con 0x7f694c15bdd0 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f695456a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c0b6100 con 0x7f694c0d1c70 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f695456a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c0b63e0 con 0x7f694c0b3370 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c0cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37378/0 (socket says 172.21.15.89:37378) 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 -- 172.21.15.89:0/2127206691 learned_addr learned my addr 172.21.15.89:0/2127206691 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952306700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0b3370 0x7f694c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6951b05700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f694c0d1c70 0x7f694c157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 -- 172.21.15.89:0/2127206691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0b3370 msgr2=0x7f694c0c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0b3370 0x7f694c0c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 -- 172.21.15.89:0/2127206691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f694c0d1c70 msgr2=0x7f694c157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f694c0d1c70 0x7f694c157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 -- 172.21.15.89:0/2127206691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69480094a0 con 0x7f694c15bdd0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.165+0000 7f6952b07700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c0cdc70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f694400cef0 tx=0x7f694400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.166+0000 7f6942ffd700 1 -- 172.21.15.89:0/2127206691 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694400ad00 con 0x7f694c15bdd0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.166+0000 7f6942ffd700 1 -- 172.21.15.89:0/2127206691 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694400ae60 con 0x7f694c15bdd0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.166+0000 7f6942ffd700 1 -- 172.21.15.89:0/2127206691 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944015680 con 0x7f694c15bdd0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.166+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c0b66c0 con 0x7f694c15bdd0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.166+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694c0b6c80 con 0x7f694c15bdd0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.167+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694c067f60 con 0x7f694c15bdd0 2023-11-24T06:20:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.167+0000 7f6942ffd700 1 -- 172.21.15.89:0/2127206691 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f694400d430 con 0x7f694c15bdd0 2023-11-24T06:20:06.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.167+0000 7f6942ffd700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f693006aed0 0x7f693006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.168+0000 7f6942ffd700 1 -- 172.21.15.89:0/2127206691 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6944089a70 con 0x7f694c15bdd0 2023-11-24T06:20:06.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.168+0000 7f6952306700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f693006aed0 0x7f693006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.168+0000 7f6952306700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f693006aed0 0x7f693006d390 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f69480090b0 tx=0x7f694801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:06.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.172+0000 7f6942ffd700 1 -- 172.21.15.89:0/2127206691 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6944054b50 con 0x7f694c15bdd0 2023-11-24T06:20:06.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.378+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f694c0b7c60 con 0x7f694c15bdd0 2023-11-24T06:20:06.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.379+0000 7f6942ffd700 1 -- 172.21.15.89:0/2127206691 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) v1 ==== 80+0+122 (secure 0 0 0) 0x7f6944058170 con 0x7f694c15bdd0 2023-11-24T06:20:06.382 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6837/2053513777 2023-11-25T06:19:59.514768+0000 2023-11-24T06:20:06.382 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6835/2053513777 2023-11-25T06:19:59.514768+0000 2023-11-24T06:20:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f693006aed0 msgr2=0x7f693006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f693006aed0 0x7f693006d390 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f69480090b0 tx=0x7f694801c040 comp rx=0 tx=0).stop 2023-11-24T06:20:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 msgr2=0x7f694c0cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c0cdc70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f694400cef0 tx=0x7f694400cf20 comp rx=0 tx=0).stop 2023-11-24T06:20:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 shutdown_connections 2023-11-24T06:20:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f694c15bdd0 0x7f694c0cdc70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f694c0b3370 0x7f694c0c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f693006aed0 0x7f693006d390 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 --2- 172.21.15.89:0/2127206691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f694c0d1c70 0x7f694c157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 >> 172.21.15.89:0/2127206691 conn(0x7f694c0731a0 msgr2=0x7f694c0b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 shutdown_connections 2023-11-24T06:20:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.381+0000 7f695456a700 1 -- 172.21.15.89:0/2127206691 wait complete. 2023-11-24T06:20:06.385 INFO:teuthology.orchestra.run.smithi089.stderr:listed 2 entries 2023-11-24T06:20:06.395 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T06:20:06.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:06.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T06:20:06.447 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:20:06.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:06.448 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T06:20:06.469 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:06.469 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:06.486 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:20:06.486 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:06.486 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:06.529 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:20:06.530 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:06.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.647+0000 7f0dcb06e700 1 Processor -- start 2023-11-24T06:20:06.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.647+0000 7f0dcb06e700 1 -- start start 2023-11-24T06:20:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dcb06e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc41446a0 0x7f0dc40b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dcb06e700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40af950 con 0x7f0dc406c710 2023-11-24T06:20:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dcb06e700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40afc30 con 0x7f0dc40adf00 2023-11-24T06:20:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dcb06e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4144a80 con 0x7f0dc41446a0 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc960b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0dc406c710 0x7f0dc40b3d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40890/0 (socket says 172.21.15.89:40890) 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc960b700 1 -- 172.21.15.89:0/4078574201 learned_addr learned my addr 172.21.15.89:0/4078574201 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709024677 0 0) 0x7f0dc40af950 con 0x7f0dc406c710 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc8e0a700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc41446a0 0x7f0dc40b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db40057c0 con 0x7f0dc406c710 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 828784225 0 0) 0x7f0db40057c0 con 0x7f0dc406c710 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc41446a0 msgr2=0x7f0dc40b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc41446a0 0x7f0dc40b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 >> v1:172.21.15.123:6789/0 conn(0x7f0dc40adf00 legacy=0x7f0dc406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc40aff10 con 0x7f0dc406c710 2023-11-24T06:20:06.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f0dbc0019f0 con 0x7f0dc406c710 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 >> v1:172.21.15.89:6789/0 conn(0x7f0dc406c710 legacy=0x7f0dc40b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.648+0000 7f0dc37fe700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0db4005c40 0x7f0db4006060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc37fe700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0db4006a60 0x7f0db4006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc37fe700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0db400dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4144a80 con 0x7f0db4007420 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db400e2f0 con 0x7f0db4006a60 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db400e4b0 con 0x7f0db4005c40 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc8e0a700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc41446a0 0x7f0dc40b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc960b700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0db400dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc3fff700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0db4006a60 0x7f0db4006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc8e0a700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0db4005c40 0x7f0db4006060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc960b700 1 -- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0db4005c40 msgr2=0x7f0db4006060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc960b700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0db4005c40 0x7f0db4006060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc960b700 1 -- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0db4006a60 msgr2=0x7f0db4006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc960b700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0db4006a60 0x7f0db4006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc960b700 1 -- 172.21.15.89:0/4078574201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db40057c0 con 0x7f0db4007420 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc960b700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0db400dc20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00a830 tx=0x7f0db4007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b888e4ef64c3fca server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc00a2f0 con 0x7f0db4007420 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc012620 con 0x7f0db4007420 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.649+0000 7f0dc37fe700 1 -- 172.21.15.89:0/4078574201 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0129d0 con 0x7f0db4007420 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.650+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4078574201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 msgr2=0x7f0db400dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.650+0000 7f0dcb06e700 1 --2- 172.21.15.89:0/4078574201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0db400dc20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00a830 tx=0x7f0db4007a90 comp rx=0 tx=0).stop 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.650+0000 7f0dc960b700 1 -- 172.21.15.89:0/4078574201 reap_dead start 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.650+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4078574201 shutdown_connections 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.650+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4078574201 >> 172.21.15.89:0/4078574201 conn(0x7f0dc4071b90 msgr2=0x7f0dc4071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.650+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4078574201 shutdown_connections 2023-11-24T06:20:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.650+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4078574201 wait complete. 2023-11-24T06:20:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.651+0000 7f0dcb06e700 1 Processor -- start 2023-11-24T06:20:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.651+0000 7f0dcb06e700 1 -- start start 2023-11-24T06:20:06.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dcb06e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0dc413ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dcb06e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc4149170 0x7f0dc40cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dcb06e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc413c860 0x7f0dc4142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dcb06e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc414bd40 con 0x7f0db4007420 2023-11-24T06:20:06.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dcb06e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40cdb40 con 0x7f0dc4149170 2023-11-24T06:20:06.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dcb06e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc414bfe0 con 0x7f0dc413c860 2023-11-24T06:20:06.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0dc413ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0dc413ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37394/0 (socket says 172.21.15.89:37394) 2023-11-24T06:20:06.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 -- 172.21.15.89:0/4147428021 learned_addr learned my addr 172.21.15.89:0/4147428021 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:06.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 -- 172.21.15.89:0/4147428021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc413c860 msgr2=0x7f0dc4142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc413c860 0x7f0dc4142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 -- 172.21.15.89:0/4147428021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc4149170 msgr2=0x7f0dc40cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc4149170 0x7f0dc40cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 -- 172.21.15.89:0/4147428021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc40cf320 con 0x7f0db4007420 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc8e0a700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0dc413ff90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0db80107c0 tx=0x7f0db8010b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.652+0000 7f0dc17fa700 1 -- 172.21.15.89:0/4147428021 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8004230 con 0x7f0db4007420 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.653+0000 7f0dc17fa700 1 -- 172.21.15.89:0/4147428021 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db800c050 con 0x7f0db4007420 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.653+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc40cf600 con 0x7f0db4007420 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.653+0000 7f0dc17fa700 1 -- 172.21.15.89:0/4147428021 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db80036a0 con 0x7f0db4007420 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.653+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc413cd10 con 0x7f0db4007420 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.654+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc40753f0 con 0x7f0db4007420 2023-11-24T06:20:06.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.654+0000 7f0dc17fa700 1 -- 172.21.15.89:0/4147428021 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0db8003800 con 0x7f0db4007420 2023-11-24T06:20:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.654+0000 7f0dc17fa700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0da806aed0 0x7f0da806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.655+0000 7f0dc3fff700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0da806aed0 0x7f0da806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.655+0000 7f0dc17fa700 1 -- 172.21.15.89:0/4147428021 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0db8089ca0 con 0x7f0db4007420 2023-11-24T06:20:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.655+0000 7f0dc3fff700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0da806aed0 0x7f0da806d390 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4149590 tx=0x7f0db0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.659+0000 7f0dc17fa700 1 -- 172.21.15.89:0/4147428021 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0db8054d00 con 0x7f0db4007420 2023-11-24T06:20:06.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.891+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dc4067f60 con 0x7f0db4007420 2023-11-24T06:20:06.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.891+0000 7f0dc17fa700 1 -- 172.21.15.89:0/4147428021 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f0db8058320 con 0x7f0db4007420 2023-11-24T06:20:06.895 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:06.895 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/503786989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":503786989},{"type":"v1","addr":"172.21.15.123:6834","nonce":503786989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3511736853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3511736853},{"type":"v1","addr":"172.21.15.89:6836","nonce":3511736853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5361,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3128631650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3128631650},{"type":"v1","addr":"172.21.15.89:6837","nonce":3128631650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:19:43.989850+0000","modified":"2023-11-24T06:19:47.380732+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6835/1601724762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1601724762},{"type":"v1","addr":"172.21.15.123:6835","nonce":1601724762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:20:06.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0da806aed0 msgr2=0x7f0da806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0da806aed0 0x7f0da806d390 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4149590 tx=0x7f0db0009040 comp rx=0 tx=0).stop 2023-11-24T06:20:06.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 msgr2=0x7f0dc413ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:06.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0dc413ff90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0db80107c0 tx=0x7f0db8010b80 comp rx=0 tx=0).stop 2023-11-24T06:20:06.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 shutdown_connections 2023-11-24T06:20:06.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc413c860 0x7f0dc4142fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db4007420 0x7f0dc413ff90 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0da806aed0 0x7f0da806d390 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 --2- 172.21.15.89:0/4147428021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc4149170 0x7f0dc40cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:06.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 >> 172.21.15.89:0/4147428021 conn(0x7f0dc4071b90 msgr2=0x7f0dc40b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:06.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 shutdown_connections 2023-11-24T06:20:06.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:06.894+0000 7f0dcb06e700 1 -- 172.21.15.89:0/4147428021 wait complete. 2023-11-24T06:20:06.899 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-11-24T06:20:06.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:07.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.026+0000 7f6e478aa700 1 Processor -- start 2023-11-24T06:20:07.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.026+0000 7f6e478aa700 1 -- start start 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e478aa700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6e4015e080 0x7f6e400b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e478aa700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e400af4d0 0x7f6e400afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e478aa700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e400ae430 con 0x7f6e4006c000 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e478aa700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e400af0b0 con 0x7f6e4015e080 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e478aa700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e400b05a0 con 0x7f6e400af4d0 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e45646700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6e4015e080 0x7f6e400b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e44e45700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e400af4d0 0x7f6e400afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e45e47700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f6e4006c000 0x7f6e40073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:44732/0 (socket says 172.21.15.89:44732) 2023-11-24T06:20:07.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e45646700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6e4015e080 0x7f6e400b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:52514/0 (socket says 172.21.15.89:52514) 2023-11-24T06:20:07.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e45e47700 1 -- 172.21.15.89:0/453594028 learned_addr learned my addr 172.21.15.89:0/453594028 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:07.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e44e45700 1 -- 172.21.15.89:0/453594028 >> v2:172.21.15.123:3300/0 conn(0x7f6e4015e080 msgr2=0x7f6e400b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:07.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e44e45700 1 --2- 172.21.15.89:0/453594028 >> v2:172.21.15.123:3300/0 conn(0x7f6e4015e080 0x7f6e400b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e44e45700 1 -- 172.21.15.89:0/453594028 >> v1:172.21.15.123:6789/0 conn(0x7f6e4006c000 legacy=0x7f6e40073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e44e45700 1 -- 172.21.15.89:0/453594028 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e400ae710 con 0x7f6e400af4d0 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.027+0000 7f6e45646700 1 --2- 172.21.15.89:0/453594028 >> v2:172.21.15.123:3300/0 conn(0x7f6e4015e080 0x7f6e400b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e44e45700 1 --2- 172.21.15.89:0/453594028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e400af4d0 0x7f6e400afb90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009840 tx=0x7f6e3c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f13b6d76a4351155 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e37fff700 1 -- 172.21.15.89:0/453594028 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c011830 con 0x7f6e400af4d0 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e37fff700 1 -- 172.21.15.89:0/453594028 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c011990 con 0x7f6e400af4d0 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e37fff700 1 -- 172.21.15.89:0/453594028 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c011d20 con 0x7f6e400af4d0 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e478aa700 1 -- 172.21.15.89:0/453594028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e400af4d0 msgr2=0x7f6e400afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e478aa700 1 --2- 172.21.15.89:0/453594028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e400af4d0 0x7f6e400afb90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009840 tx=0x7f6e3c009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e478aa700 1 -- 172.21.15.89:0/453594028 shutdown_connections 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e478aa700 1 --2- 172.21.15.89:0/453594028 >> v2:172.21.15.123:3300/0 conn(0x7f6e4015e080 0x7f6e400b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e478aa700 1 --2- 172.21.15.89:0/453594028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e400af4d0 0x7f6e400afb90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e478aa700 1 -- 172.21.15.89:0/453594028 >> 172.21.15.89:0/453594028 conn(0x7f6e40071b90 msgr2=0x7f6e40071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:07.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e478aa700 1 -- 172.21.15.89:0/453594028 shutdown_connections 2023-11-24T06:20:07.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.028+0000 7f6e478aa700 1 -- 172.21.15.89:0/453594028 wait complete. 2023-11-24T06:20:07.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e478aa700 1 Processor -- start 2023-11-24T06:20:07.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e478aa700 1 -- start start 2023-11-24T06:20:07.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e478aa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e4006c000 0x7f6e401464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e478aa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e400af4d0 0x7f6e40157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e478aa700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e4015e080 0x7f6e40159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e478aa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40154eb0 con 0x7f6e4006c000 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e478aa700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40158860 con 0x7f6e400af4d0 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e478aa700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40146a00 con 0x7f6e4015e080 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e45646700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e4006c000 0x7f6e401464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.030+0000 7f6e45646700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e4006c000 0x7f6e401464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37408/0 (socket says 172.21.15.89:37408) 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e45646700 1 -- 172.21.15.89:0/4116892806 learned_addr learned my addr 172.21.15.89:0/4116892806 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e44e45700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e400af4d0 0x7f6e40157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e45e47700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e4015e080 0x7f6e40159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e45646700 1 -- 172.21.15.89:0/4116892806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e4015e080 msgr2=0x7f6e40159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e45646700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e4015e080 0x7f6e40159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e45646700 1 -- 172.21.15.89:0/4116892806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e400af4d0 msgr2=0x7f6e40157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e45646700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e400af4d0 0x7f6e40157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e45646700 1 -- 172.21.15.89:0/4116892806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e300094a0 con 0x7f6e4006c000 2023-11-24T06:20:07.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e45646700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e4006c000 0x7f6e401464c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000ec10 tx=0x7f6e3000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e35ffb700 1 -- 172.21.15.89:0/4116892806 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30009760 con 0x7f6e4006c000 2023-11-24T06:20:07.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e35ffb700 1 -- 172.21.15.89:0/4116892806 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3000c850 con 0x7f6e4006c000 2023-11-24T06:20:07.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3c0094a0 con 0x7f6e4006c000 2023-11-24T06:20:07.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e35ffb700 1 -- 172.21.15.89:0/4116892806 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3000cbe0 con 0x7f6e4006c000 2023-11-24T06:20:07.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.031+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e40155090 con 0x7f6e4006c000 2023-11-24T06:20:07.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.032+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e400753f0 con 0x7f6e4006c000 2023-11-24T06:20:07.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.033+0000 7f6e35ffb700 1 -- 172.21.15.89:0/4116892806 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6e3000cd40 con 0x7f6e4006c000 2023-11-24T06:20:07.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.033+0000 7f6e35ffb700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e2406af00 0x7f6e2406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.033+0000 7f6e35ffb700 1 -- 172.21.15.89:0/4116892806 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6e30089620 con 0x7f6e4006c000 2023-11-24T06:20:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.033+0000 7f6e44e45700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e2406af00 0x7f6e2406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.033+0000 7f6e44e45700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e2406af00 0x7f6e2406d3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0090b0 tx=0x7f6e3c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.037+0000 7f6e35ffb700 1 -- 172.21.15.89:0/4116892806 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e300545e0 con 0x7f6e4006c000 2023-11-24T06:20:07.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.227+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e40067f60 con 0x7f6e4006c000 2023-11-24T06:20:07.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.228+0000 7f6e35ffb700 1 -- 172.21.15.89:0/4116892806 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16662 (secure 0 0 0) 0x7f6e30057c00 con 0x7f6e4006c000 2023-11-24T06:20:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:07.232 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:02.530677+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/2053513777":"2023-11-25T06:19:59.514768+0000","172.21.15.89:6835/2053513777":"2023-11-25T06:19:59.514768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.230+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e2406af00 msgr2=0x7f6e2406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.230+0000 7f6e478aa700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e2406af00 0x7f6e2406d3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0090b0 tx=0x7f6e3c008030 comp rx=0 tx=0).stop 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e4006c000 msgr2=0x7f6e401464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e4006c000 0x7f6e401464c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000ec10 tx=0x7f6e3000ef20 comp rx=0 tx=0).stop 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 shutdown_connections 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e4015e080 0x7f6e40159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e4006c000 0x7f6e401464c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6e2406af00 0x7f6e2406d3c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 --2- 172.21.15.89:0/4116892806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e400af4d0 0x7f6e40157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 >> 172.21.15.89:0/4116892806 conn(0x7f6e40071b90 msgr2=0x7f6e400d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:07.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 shutdown_connections 2023-11-24T06:20:07.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.231+0000 7f6e478aa700 1 -- 172.21.15.89:0/4116892806 wait complete. 2023-11-24T06:20:07.246 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:19:41.783017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:19:42.953803+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:07.246 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:07.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.367+0000 7f0d097c0700 1 Processor -- start 2023-11-24T06:20:07.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.367+0000 7f0d097c0700 1 -- start start 2023-11-24T06:20:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.367+0000 7f0d097c0700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d04151660 0x7f0d040cf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.367+0000 7f0d097c0700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d0406dec0 0x7f0d04070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.367+0000 7f0d097c0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 0x7f0d0415e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.367+0000 7f0d097c0700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0406ee40 con 0x7f0d04151660 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.367+0000 7f0d097c0700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d040b2b00 con 0x7f0d0406dec0 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.367+0000 7f0d097c0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0406e3a0 con 0x7f0d04071080 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d04151660 0x7f0d040cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d04151660 0x7f0d040cf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37416/0 (socket says 172.21.15.89:37416) 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 -- 172.21.15.89:0/2081616561 learned_addr learned my addr 172.21.15.89:0/2081616561 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d037fe700 1 --2- 172.21.15.89:0/2081616561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 0x7f0d0415e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d027fc700 1 --2- 172.21.15.89:0/2081616561 >> v2:172.21.15.123:3300/0 conn(0x7f0d0406dec0 0x7f0d04070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 -- 172.21.15.89:0/2081616561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 msgr2=0x7f0d0415e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 --2- 172.21.15.89:0/2081616561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 0x7f0d0415e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 -- 172.21.15.89:0/2081616561 >> v2:172.21.15.123:3300/0 conn(0x7f0d0406dec0 msgr2=0x7f0d04070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 --2- 172.21.15.89:0/2081616561 >> v2:172.21.15.123:3300/0 conn(0x7f0d0406dec0 0x7f0d04070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 -- 172.21.15.89:0/2081616561 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d040b2da0 con 0x7f0d04151660 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d02ffd700 1 --2- 172.21.15.89:0/2081616561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d040cf300 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0d040b4950 tx=0x7f0cf800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5568d2fd544655b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d01ffb700 1 -- 172.21.15.89:0/2081616561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf800dd30 con 0x7f0d04151660 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d01ffb700 1 -- 172.21.15.89:0/2081616561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf800de90 con 0x7f0d04151660 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.368+0000 7f0d01ffb700 1 -- 172.21.15.89:0/2081616561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8010690 con 0x7f0d04151660 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 -- 172.21.15.89:0/2081616561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 msgr2=0x7f0d040cf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 --2- 172.21.15.89:0/2081616561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d040cf300 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0d040b4950 tx=0x7f0cf800be70 comp rx=0 tx=0).stop 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 -- 172.21.15.89:0/2081616561 shutdown_connections 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 --2- 172.21.15.89:0/2081616561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 0x7f0d0415e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 --2- 172.21.15.89:0/2081616561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d040cf300 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 --2- 172.21.15.89:0/2081616561 >> v2:172.21.15.123:3300/0 conn(0x7f0d0406dec0 0x7f0d04070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 -- 172.21.15.89:0/2081616561 >> 172.21.15.89:0/2081616561 conn(0x7f0d0406b410 msgr2=0x7f0d0406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:07.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 -- 172.21.15.89:0/2081616561 shutdown_connections 2023-11-24T06:20:07.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.369+0000 7f0d097c0700 1 -- 172.21.15.89:0/2081616561 wait complete. 2023-11-24T06:20:07.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.370+0000 7f0d097c0700 1 Processor -- start 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.371+0000 7f0d097c0700 1 -- start start 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.371+0000 7f0d097c0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d0406dec0 0x7f0d041e5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.371+0000 7f0d097c0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 0x7f0d04145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.371+0000 7f0d097c0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d041463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.371+0000 7f0d097c0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0415c160 con 0x7f0d04151660 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.371+0000 7f0d097c0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d041e4380 con 0x7f0d0406dec0 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.371+0000 7f0d097c0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d041e4660 con 0x7f0d04071080 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d041463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d041463f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37426/0 (socket says 172.21.15.89:37426) 2023-11-24T06:20:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 -- 172.21.15.89:0/1465544911 learned_addr learned my addr 172.21.15.89:0/1465544911 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d02ffd700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d0406dec0 0x7f0d041e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d027fc700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 0x7f0d04145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 -- 172.21.15.89:0/1465544911 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 msgr2=0x7f0d04145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 0x7f0d04145610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 -- 172.21.15.89:0/1465544911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d0406dec0 msgr2=0x7f0d041e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d0406dec0 0x7f0d041e5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 -- 172.21.15.89:0/1465544911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf800b7f0 con 0x7f0d04151660 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d037fe700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d041463f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc007e40 tx=0x7f0cfc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0ceb7fe700 1 -- 172.21.15.89:0/1465544911 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc00ed30 con 0x7f0d04151660 2023-11-24T06:20:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0ceb7fe700 1 -- 172.21.15.89:0/1465544911 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc014420 con 0x7f0d04151660 2023-11-24T06:20:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d041e4940 con 0x7f0d04151660 2023-11-24T06:20:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0ceb7fe700 1 -- 172.21.15.89:0/1465544911 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc0147d0 con 0x7f0d04151660 2023-11-24T06:20:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.372+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d041e5230 con 0x7f0d04151660 2023-11-24T06:20:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.373+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0415c460 con 0x7f0d04151660 2023-11-24T06:20:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.374+0000 7f0ceb7fe700 1 -- 172.21.15.89:0/1465544911 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0cfc014930 con 0x7f0d04151660 2023-11-24T06:20:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.374+0000 7f0ceb7fe700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0cf006aed0 0x7f0cf006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.374+0000 7f0d02ffd700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0cf006aed0 0x7f0cf006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.374+0000 7f0ceb7fe700 1 -- 172.21.15.89:0/1465544911 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0cfc089c60 con 0x7f0d04151660 2023-11-24T06:20:07.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.375+0000 7f0d02ffd700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0cf006aed0 0x7f0cf006d390 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800beb0 tx=0x7f0cf8012000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.379+0000 7f0ceb7fe700 1 -- 172.21.15.89:0/1465544911 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cfc054d40 con 0x7f0d04151660 2023-11-24T06:20:07.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.569+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d04068480 con 0x7f0d04151660 2023-11-24T06:20:07.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.569+0000 7f0ceb7fe700 1 -- 172.21.15.89:0/1465544911 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0cfc058360 con 0x7f0d04151660 2023-11-24T06:20:07.573 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:07.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0cf006aed0 msgr2=0x7f0cf006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0cf006aed0 0x7f0cf006d390 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800beb0 tx=0x7f0cf8012000 comp rx=0 tx=0).stop 2023-11-24T06:20:07.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 msgr2=0x7f0d041463f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d041463f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc007e40 tx=0x7f0cfc00e3c0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 shutdown_connections 2023-11-24T06:20:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04151660 0x7f0d041463f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0cf006aed0 0x7f0cf006d390 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d0406dec0 0x7f0d041e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 --2- 172.21.15.89:0/1465544911 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d04071080 0x7f0d04145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 >> 172.21.15.89:0/1465544911 conn(0x7f0d0406b410 msgr2=0x7f0d040b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 shutdown_connections 2023-11-24T06:20:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.572+0000 7f0d097c0700 1 -- 172.21.15.89:0/1465544911 wait complete. 2023-11-24T06:20:07.587 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:20:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.707+0000 7fe6de1ed700 1 Processor -- start 2023-11-24T06:20:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.707+0000 7fe6de1ed700 1 -- start start 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.708+0000 7fe6de1ed700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6d815e250 0x7fe6d8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.708+0000 7fe6de1ed700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d8162ad0 con 0x7fe6d80afea0 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.708+0000 7fe6de1ed700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d80b48f0 con 0x7fe6d80b0560 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.708+0000 7fe6de1ed700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d815e630 con 0x7fe6d815e250 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.708+0000 7fe6d7fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe6d80afea0 0x7fe6d81623c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40938/0 (socket says 172.21.15.89:40938) 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.708+0000 7fe6d7fff700 1 -- 172.21.15.89:0/1127848281 learned_addr learned my addr 172.21.15.89:0/1127848281 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.708+0000 7fe6d77fe700 1 --2- 172.21.15.89:0/1127848281 >> v2:172.21.15.89:3300/0 conn(0x7fe6d815e250 0x7fe6d8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d67fc700 1 -- 172.21.15.89:0/1127848281 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152041382 0 0) 0x7fe6d8162ad0 con 0x7fe6d80afea0 2023-11-24T06:20:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d67fc700 1 -- 172.21.15.89:0/1127848281 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c00035b0 con 0x7fe6d80afea0 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d77fe700 1 -- 172.21.15.89:0/1127848281 >> v1:172.21.15.123:6789/0 conn(0x7fe6d80b0560 legacy=0x7fe6d8161cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d77fe700 1 -- 172.21.15.89:0/1127848281 >> v1:172.21.15.89:6789/0 conn(0x7fe6d80afea0 legacy=0x7fe6d81623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d77fe700 1 -- 172.21.15.89:0/1127848281 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80b4bd0 con 0x7fe6d815e250 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d77fe700 1 --2- 172.21.15.89:0/1127848281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d815e250 0x7fe6d8073f90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009840 tx=0x7fe6cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a006240266361fe9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d67fc700 1 -- 172.21.15.89:0/1127848281 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc011830 con 0x7fe6d815e250 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d67fc700 1 -- 172.21.15.89:0/1127848281 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc011990 con 0x7fe6d815e250 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.709+0000 7fe6d67fc700 1 -- 172.21.15.89:0/1127848281 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc011d20 con 0x7fe6d815e250 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.710+0000 7fe6de1ed700 1 -- 172.21.15.89:0/1127848281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d815e250 msgr2=0x7fe6d8073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.710+0000 7fe6de1ed700 1 --2- 172.21.15.89:0/1127848281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d815e250 0x7fe6d8073f90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009840 tx=0x7fe6cc009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.710+0000 7fe6de1ed700 1 -- 172.21.15.89:0/1127848281 shutdown_connections 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.710+0000 7fe6de1ed700 1 --2- 172.21.15.89:0/1127848281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d815e250 0x7fe6d8073f90 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.710+0000 7fe6de1ed700 1 -- 172.21.15.89:0/1127848281 >> 172.21.15.89:0/1127848281 conn(0x7fe6d8072960 msgr2=0x7fe6d806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.710+0000 7fe6de1ed700 1 -- 172.21.15.89:0/1127848281 shutdown_connections 2023-11-24T06:20:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.710+0000 7fe6de1ed700 1 -- 172.21.15.89:0/1127848281 wait complete. 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.711+0000 7fe6de1ed700 1 Processor -- start 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.711+0000 7fe6de1ed700 1 -- start start 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6de1ed700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6d80afea0 0x7fe6d813f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6de1ed700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d80b0560 0x7fe6d80b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6de1ed700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6d815e250 0x7fe6d80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6de1ed700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80ce190 con 0x7fe6d80b0560 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6de1ed700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d813fb80 con 0x7fe6d80afea0 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6de1ed700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80cd7f0 con 0x7fe6d815e250 2023-11-24T06:20:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d80b0560 0x7fe6d80b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d80b0560 0x7fe6d80b1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37438/0 (socket says 172.21.15.89:37438) 2023-11-24T06:20:07.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 -- 172.21.15.89:0/3141515429 learned_addr learned my addr 172.21.15.89:0/3141515429 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:07.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d77fe700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6d80afea0 0x7fe6d813f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d7fff700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6d815e250 0x7fe6d80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 -- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6d815e250 msgr2=0x7fe6d80b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6d815e250 0x7fe6d80b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 -- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6d80afea0 msgr2=0x7fe6d813f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6d80afea0 0x7fe6d813f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 -- 172.21.15.89:0/3141515429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc0094a0 con 0x7fe6d80b0560 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d7fff700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6d815e250 0x7fe6d80b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6d6ffd700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d80b0560 0x7fe6d80b1cd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800c4b0 tx=0x7fe6c800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6bffff700 1 -- 172.21.15.89:0/3141515429 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c80052d0 con 0x7fe6d80b0560 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6bffff700 1 -- 172.21.15.89:0/3141515429 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c8005430 con 0x7fe6d80b0560 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d80cdad0 con 0x7fe6d80b0560 2023-11-24T06:20:07.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.712+0000 7fe6bffff700 1 -- 172.21.15.89:0/3141515429 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c80038c0 con 0x7fe6d80b0560 2023-11-24T06:20:07.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.713+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d8161570 con 0x7fe6d80b0560 2023-11-24T06:20:07.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.713+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d80ce2d0 con 0x7fe6d80b0560 2023-11-24T06:20:07.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.714+0000 7fe6bffff700 1 -- 172.21.15.89:0/3141515429 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe6c8003a20 con 0x7fe6d80b0560 2023-11-24T06:20:07.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.714+0000 7fe6bffff700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c406aed0 0x7fe6c406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:07.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.714+0000 7fe6bffff700 1 -- 172.21.15.89:0/3141515429 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe6c8088a70 con 0x7fe6d80b0560 2023-11-24T06:20:07.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.714+0000 7fe6d77fe700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c406aed0 0x7fe6c406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:07.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.715+0000 7fe6d77fe700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c406aed0 0x7fe6c406d390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0090b0 tx=0x7fe6cc01c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:07.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.719+0000 7fe6bffff700 1 -- 172.21.15.89:0/3141515429 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6d80ce2d0 con 0x7fe6d80b0560 2023-11-24T06:20:07.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.910+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe6d80ce2d0 con 0x7fe6d80b0560 2023-11-24T06:20:07.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.910+0000 7fe6bffff700 1 -- 172.21.15.89:0/3141515429 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe6c8017020 con 0x7fe6d80b0560 2023-11-24T06:20:07.913 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:20:07.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c406aed0 msgr2=0x7fe6c406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c406aed0 0x7fe6c406d390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0090b0 tx=0x7fe6cc01c040 comp rx=0 tx=0).stop 2023-11-24T06:20:07.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d80b0560 msgr2=0x7fe6d80b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:07.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d80b0560 0x7fe6d80b1cd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800c4b0 tx=0x7fe6c800c7c0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 shutdown_connections 2023-11-24T06:20:07.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6d815e250 0x7fe6d80b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe6c406aed0 0x7fe6c406d390 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6d80afea0 0x7fe6d813f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 --2- 172.21.15.89:0/3141515429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d80b0560 0x7fe6d80b1cd0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:07.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.912+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 >> 172.21.15.89:0/3141515429 conn(0x7fe6d8072960 msgr2=0x7fe6d80af340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:07.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.913+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 shutdown_connections 2023-11-24T06:20:07.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:07.913+0000 7fe6de1ed700 1 -- 172.21.15.89:0/3141515429 wait complete. 2023-11-24T06:20:07.928 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:20:08.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.048+0000 7fb12cf00700 1 Processor -- start 2023-11-24T06:20:08.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.048+0000 7fb12cf00700 1 -- start start 2023-11-24T06:20:08.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12cf00700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb12815e160 0x7fb1280b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12cf00700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 0x7fb128073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12cf00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1280b3c60 con 0x7fb1280744d0 2023-11-24T06:20:08.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12cf00700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1280b4500 con 0x7fb12815e160 2023-11-24T06:20:08.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12cf00700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb128074960 con 0x7fb1280afc90 2023-11-24T06:20:08.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb126d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb1280744d0 0x7fb12806d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40952/0 (socket says 172.21.15.89:40952) 2023-11-24T06:20:08.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb126d9d700 1 -- 172.21.15.89:0/1692821205 learned_addr learned my addr 172.21.15.89:0/1692821205 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:08.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12659c700 1 --2- 172.21.15.89:0/1692821205 >> v2:172.21.15.123:3300/0 conn(0x7fb12815e160 0x7fb1280b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb125d9b700 1 --2- 172.21.15.89:0/1692821205 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 0x7fb128073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12559a700 1 -- 172.21.15.89:0/1692821205 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660710570 0 0) 0x7fb1280b3c60 con 0x7fb1280744d0 2023-11-24T06:20:08.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12559a700 1 -- 172.21.15.89:0/1692821205 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1100035b0 con 0x7fb1280744d0 2023-11-24T06:20:08.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12659c700 1 -- 172.21.15.89:0/1692821205 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 msgr2=0x7fb128073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:08.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12659c700 1 --2- 172.21.15.89:0/1692821205 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 0x7fb128073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12659c700 1 -- 172.21.15.89:0/1692821205 >> v1:172.21.15.89:6789/0 conn(0x7fb1280744d0 legacy=0x7fb12806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12659c700 1 -- 172.21.15.89:0/1692821205 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1280b58c0 con 0x7fb12815e160 2023-11-24T06:20:08.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.049+0000 7fb12659c700 1 --2- 172.21.15.89:0/1692821205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 0x7fb1280b5da0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009840 tx=0x7fb11c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d0065a2b6214086 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12559a700 1 -- 172.21.15.89:0/1692821205 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c011830 con 0x7fb12815e160 2023-11-24T06:20:08.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12559a700 1 -- 172.21.15.89:0/1692821205 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c011990 con 0x7fb12815e160 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12559a700 1 -- 172.21.15.89:0/1692821205 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c011d00 con 0x7fb12815e160 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12cf00700 1 -- 172.21.15.89:0/1692821205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 msgr2=0x7fb1280b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12cf00700 1 --2- 172.21.15.89:0/1692821205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 0x7fb1280b5da0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009840 tx=0x7fb11c009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12cf00700 1 -- 172.21.15.89:0/1692821205 shutdown_connections 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12cf00700 1 --2- 172.21.15.89:0/1692821205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 0x7fb1280b5da0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12cf00700 1 --2- 172.21.15.89:0/1692821205 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 0x7fb128073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12cf00700 1 -- 172.21.15.89:0/1692821205 >> 172.21.15.89:0/1692821205 conn(0x7fb1280d0bf0 msgr2=0x7fb1280d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12cf00700 1 -- 172.21.15.89:0/1692821205 shutdown_connections 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.050+0000 7fb12cf00700 1 -- 172.21.15.89:0/1692821205 wait complete. 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12cf00700 1 Processor -- start 2023-11-24T06:20:08.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12cf00700 1 -- start start 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12cf00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280744d0 0x7fb1281f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12cf00700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 0x7fb1280b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12cf00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 0x7fb1280b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12cf00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12814d610 con 0x7fb1280744d0 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12cf00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12814d330 con 0x7fb12815e160 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12cf00700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1281f70a0 con 0x7fb1280afc90 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280744d0 0x7fb1281f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280744d0 0x7fb1281f3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37452/0 (socket says 172.21.15.89:37452) 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.052+0000 7fb12659c700 1 -- 172.21.15.89:0/953772405 learned_addr learned my addr 172.21.15.89:0/953772405 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb126d9d700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 0x7fb1280b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb125d9b700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 0x7fb1280b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb12659c700 1 -- 172.21.15.89:0/953772405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 msgr2=0x7fb1280b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb12659c700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 0x7fb1280b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb12659c700 1 -- 172.21.15.89:0/953772405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 msgr2=0x7fb1280b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb12659c700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 0x7fb1280b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb12659c700 1 -- 172.21.15.89:0/953772405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0094a0 con 0x7fb1280744d0 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb12659c700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280744d0 0x7fb1281f3050 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb11c0090b0 tx=0x7fb11c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb116ffd700 1 -- 172.21.15.89:0/953772405 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c004550 con 0x7fb1280744d0 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb116ffd700 1 -- 172.21.15.89:0/953772405 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c01e070 con 0x7fb1280744d0 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb116ffd700 1 -- 172.21.15.89:0/953772405 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0038e0 con 0x7fb1280744d0 2023-11-24T06:20:08.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1281f7380 con 0x7fb1280744d0 2023-11-24T06:20:08.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.053+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1281f3590 con 0x7fb1280744d0 2023-11-24T06:20:08.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.054+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1281f6f00 con 0x7fb1280744d0 2023-11-24T06:20:08.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.054+0000 7fb116ffd700 1 -- 172.21.15.89:0/953772405 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb11c003a40 con 0x7fb1280744d0 2023-11-24T06:20:08.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.055+0000 7fb116ffd700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb10806aed0 0x7fb10806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.055+0000 7fb116ffd700 1 -- 172.21.15.89:0/953772405 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb11c08a3f0 con 0x7fb1280744d0 2023-11-24T06:20:08.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.055+0000 7fb125d9b700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb10806aed0 0x7fb10806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.055+0000 7fb125d9b700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb10806aed0 0x7fb10806d390 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb118009be0 tx=0x7fb118009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.059+0000 7fb116ffd700 1 -- 172.21.15.89:0/953772405 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1281f6f00 con 0x7fb1280744d0 2023-11-24T06:20:08.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.247+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb1281f7840 con 0x7fb1280744d0 2023-11-24T06:20:08.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.248+0000 7fb116ffd700 1 -- 172.21.15.89:0/953772405 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb11c020020 con 0x7fb1280744d0 2023-11-24T06:20:08.251 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:20:08.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb10806aed0 msgr2=0x7fb10806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb10806aed0 0x7fb10806d390 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb118009be0 tx=0x7fb118009210 comp rx=0 tx=0).stop 2023-11-24T06:20:08.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280744d0 msgr2=0x7fb1281f3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280744d0 0x7fb1281f3050 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb11c0090b0 tx=0x7fb11c004750 comp rx=0 tx=0).stop 2023-11-24T06:20:08.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 shutdown_connections 2023-11-24T06:20:08.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb10806aed0 0x7fb10806d390 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb12815e160 0x7fb1280b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280744d0 0x7fb1281f3050 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 --2- 172.21.15.89:0/953772405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1280afc90 0x7fb1280b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 >> 172.21.15.89:0/953772405 conn(0x7fb1280d0bf0 msgr2=0x7fb1280aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:08.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 shutdown_connections 2023-11-24T06:20:08.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.250+0000 7fb12cf00700 1 -- 172.21.15.89:0/953772405 wait complete. 2023-11-24T06:20:08.265 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:08.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.383+0000 7f58a311d700 1 Processor -- start 2023-11-24T06:20:08.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.383+0000 7f58a311d700 1 -- start start 2023-11-24T06:20:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.383+0000 7f58a311d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.383+0000 7f58a311d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f589c06c710 0x7f589c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.383+0000 7f58a311d700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589c0ae430 con 0x7f589c0af4d0 2023-11-24T06:20:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.383+0000 7f58a311d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589c0af0b0 con 0x7f589c06c710 2023-11-24T06:20:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.383+0000 7f58a311d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589c0b05a0 con 0x7f589c15e080 2023-11-24T06:20:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a16ba700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f589c06c710 0x7f589c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a0eb9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a16ba700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f589c06c710 0x7f589c0afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:52574/0 (socket says 172.21.15.89:52574) 2023-11-24T06:20:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a16ba700 1 -- 172.21.15.89:0/880543438 learned_addr learned my addr 172.21.15.89:0/880543438 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f589bfff700 1 --1- 172.21.15.89:0/880543438 >> v1:172.21.15.123:6789/0 conn(0x7f589c0af4d0 0x7f589c06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:44786/0 (socket says 172.21.15.89:44786) 2023-11-24T06:20:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a16ba700 1 -- 172.21.15.89:0/880543438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 msgr2=0x7f589c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a16ba700 1 --2- 172.21.15.89:0/880543438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a16ba700 1 -- 172.21.15.89:0/880543438 >> v1:172.21.15.123:6789/0 conn(0x7f589c0af4d0 legacy=0x7f589c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a16ba700 1 -- 172.21.15.89:0/880543438 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c0ae710 con 0x7f589c06c710 2023-11-24T06:20:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a0eb9700 1 --2- 172.21.15.89:0/880543438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.384+0000 7f58a16ba700 1 --2- 172.21.15.89:0/880543438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 0x7f589c0afb90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5894009e80 tx=0x7f589400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5423b59e9874f32 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f589b7fe700 1 -- 172.21.15.89:0/880543438 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589400e070 con 0x7f589c06c710 2023-11-24T06:20:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f589b7fe700 1 -- 172.21.15.89:0/880543438 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589400b8d0 con 0x7f589c06c710 2023-11-24T06:20:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f589b7fe700 1 -- 172.21.15.89:0/880543438 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589400bc80 con 0x7f589c06c710 2023-11-24T06:20:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f58a311d700 1 -- 172.21.15.89:0/880543438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 msgr2=0x7f589c0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f58a311d700 1 --2- 172.21.15.89:0/880543438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 0x7f589c0afb90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5894009e80 tx=0x7f589400b330 comp rx=0 tx=0).stop 2023-11-24T06:20:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f58a311d700 1 -- 172.21.15.89:0/880543438 shutdown_connections 2023-11-24T06:20:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f58a311d700 1 --2- 172.21.15.89:0/880543438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 0x7f589c0afb90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f58a311d700 1 --2- 172.21.15.89:0/880543438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f58a311d700 1 -- 172.21.15.89:0/880543438 >> 172.21.15.89:0/880543438 conn(0x7f589c071b90 msgr2=0x7f589c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f58a311d700 1 -- 172.21.15.89:0/880543438 shutdown_connections 2023-11-24T06:20:08.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.385+0000 7f58a311d700 1 -- 172.21.15.89:0/880543438 wait complete. 2023-11-24T06:20:08.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.386+0000 7f58a311d700 1 Processor -- start 2023-11-24T06:20:08.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.386+0000 7f58a311d700 1 -- start start 2023-11-24T06:20:08.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f58a311d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 0x7f589c153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f58a311d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c0af4d0 0x7f589c159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f58a311d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f58a311d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c153c80 con 0x7f589c0af4d0 2023-11-24T06:20:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f58a311d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c1f4750 con 0x7f589c06c710 2023-11-24T06:20:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f58a311d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589c1598c0 con 0x7f589c15e080 2023-11-24T06:20:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f589bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c0af4d0 0x7f589c159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f589bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c0af4d0 0x7f589c159380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37466/0 (socket says 172.21.15.89:37466) 2023-11-24T06:20:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.387+0000 7f589bfff700 1 -- 172.21.15.89:0/1857688721 learned_addr learned my addr 172.21.15.89:0/1857688721 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f58a0eb9700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 0x7f589c153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f58a16ba700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f589bfff700 1 -- 172.21.15.89:0/1857688721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 msgr2=0x7f589c06e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f589bfff700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c06e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f589bfff700 1 -- 172.21.15.89:0/1857688721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 msgr2=0x7f589c153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f589bfff700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 0x7f589c153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f589bfff700 1 -- 172.21.15.89:0/1857688721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58900094a0 con 0x7f589c0af4d0 2023-11-24T06:20:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f589bfff700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c0af4d0 0x7f589c159380 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f588800c4f0 tx=0x7f588800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f58997fa700 1 -- 172.21.15.89:0/1857688721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888005cb0 con 0x7f589c0af4d0 2023-11-24T06:20:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f58997fa700 1 -- 172.21.15.89:0/1857688721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5888005e10 con 0x7f589c0af4d0 2023-11-24T06:20:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5894009b30 con 0x7f589c0af4d0 2023-11-24T06:20:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f58997fa700 1 -- 172.21.15.89:0/1857688721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58880072d0 con 0x7f589c0af4d0 2023-11-24T06:20:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.388+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589c1f3e20 con 0x7f589c0af4d0 2023-11-24T06:20:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.389+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f589c0753f0 con 0x7f589c0af4d0 2023-11-24T06:20:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.390+0000 7f58997fa700 1 -- 172.21.15.89:0/1857688721 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5888007430 con 0x7f589c0af4d0 2023-11-24T06:20:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.390+0000 7f58997fa700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f588006aed0 0x7f588006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.390+0000 7f58997fa700 1 -- 172.21.15.89:0/1857688721 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f58880889f0 con 0x7f589c0af4d0 2023-11-24T06:20:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.390+0000 7f58a0eb9700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f588006aed0 0x7f588006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.390+0000 7f58a0eb9700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f588006aed0 0x7f588006d390 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5890009c60 tx=0x7f5890008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.394+0000 7f58997fa700 1 -- 172.21.15.89:0/1857688721 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5888053b00 con 0x7f589c0af4d0 2023-11-24T06:20:08.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.631+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f589c1f4920 con 0x7f589c0af4d0 2023-11-24T06:20:08.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.632+0000 7f58997fa700 1 -- 172.21.15.89:0/1857688721 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f589c1f4920 con 0x7f589c0af4d0 2023-11-24T06:20:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:08.635 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/503786989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":503786989},{"type":"v1","addr":"172.21.15.123:6834","nonce":503786989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3511736853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3511736853},{"type":"v1","addr":"172.21.15.89:6836","nonce":3511736853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5361,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3128631650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3128631650},{"type":"v1","addr":"172.21.15.89:6837","nonce":3128631650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:19:43.989850+0000","modified":"2023-11-24T06:19:47.380732+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6835/1601724762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1601724762},{"type":"v1","addr":"172.21.15.123:6835","nonce":1601724762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:20:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f588006aed0 msgr2=0x7f588006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f588006aed0 0x7f588006d390 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5890009c60 tx=0x7f5890008040 comp rx=0 tx=0).stop 2023-11-24T06:20:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c0af4d0 msgr2=0x7f589c159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c0af4d0 0x7f589c159380 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f588800c4f0 tx=0x7f588800c8b0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 shutdown_connections 2023-11-24T06:20:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f589c15e080 0x7f589c06e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f588006aed0 0x7f588006d390 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589c06c710 0x7f589c153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 --2- 172.21.15.89:0/1857688721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c0af4d0 0x7f589c159380 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 >> 172.21.15.89:0/1857688721 conn(0x7f589c071b90 msgr2=0x7f589c06bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 shutdown_connections 2023-11-24T06:20:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.634+0000 7f58a311d700 1 -- 172.21.15.89:0/1857688721 wait complete. 2023-11-24T06:20:08.639 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-11-24T06:20:08.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.766+0000 7f79ac53a700 1 Processor -- start 2023-11-24T06:20:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.766+0000 7f79ac53a700 1 -- start start 2023-11-24T06:20:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79ac53a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 0x7f79a41604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79ac53a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79a40b3370 0x7f79a40b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79ac53a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a40b2030 con 0x7f79a40d1c70 2023-11-24T06:20:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79ac53a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a40b2f50 con 0x7f79a40b3370 2023-11-24T06:20:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79ac53a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a40b22d0 con 0x7f79a415bdd0 2023-11-24T06:20:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a9ad5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79a40b3370 0x7f79a40b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a9ad5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79a40b3370 0x7f79a40b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37474/0 (socket says 172.21.15.89:37474) 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a9ad5700 1 -- 172.21.15.89:0/261560240 learned_addr learned my addr 172.21.15.89:0/261560240 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a9ad5700 1 -- 172.21.15.89:0/261560240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 msgr2=0x7f79a41604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79aa2d6700 1 --2- 172.21.15.89:0/261560240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 0x7f79a41604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a9ad5700 1 --2- 172.21.15.89:0/261560240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 0x7f79a41604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a9ad5700 1 -- 172.21.15.89:0/261560240 >> v1:172.21.15.123:6789/0 conn(0x7f79a40d1c70 legacy=0x7f79a40ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a9ad5700 1 -- 172.21.15.89:0/261560240 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a40b25b0 con 0x7f79a40b3370 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a9ad5700 1 --2- 172.21.15.89:0/261560240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a40b1af0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f79940098f0 tx=0x7f7994009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65e11dcba58bdac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.767+0000 7f79a92d4700 1 -- 172.21.15.89:0/261560240 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994004230 con 0x7f79a40b3370 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79a92d4700 1 -- 172.21.15.89:0/261560240 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7994004390 con 0x7f79a40b3370 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79a92d4700 1 -- 172.21.15.89:0/261560240 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79940036a0 con 0x7f79a40b3370 2023-11-24T06:20:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79ac53a700 1 -- 172.21.15.89:0/261560240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 msgr2=0x7f79a40b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79ac53a700 1 --2- 172.21.15.89:0/261560240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a40b1af0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f79940098f0 tx=0x7f7994009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79ac53a700 1 -- 172.21.15.89:0/261560240 shutdown_connections 2023-11-24T06:20:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79ac53a700 1 --2- 172.21.15.89:0/261560240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 0x7f79a41604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79ac53a700 1 --2- 172.21.15.89:0/261560240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a40b1af0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79ac53a700 1 -- 172.21.15.89:0/261560240 >> 172.21.15.89:0/261560240 conn(0x7f79a40731a0 msgr2=0x7f79a409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79ac53a700 1 -- 172.21.15.89:0/261560240 shutdown_connections 2023-11-24T06:20:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.768+0000 7f79ac53a700 1 -- 172.21.15.89:0/261560240 wait complete. 2023-11-24T06:20:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.769+0000 7f79ac53a700 1 Processor -- start 2023-11-24T06:20:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.769+0000 7f79ac53a700 1 -- start start 2023-11-24T06:20:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79ac53a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a4156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79ac53a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40d1c70 0x7f79a40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79ac53a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 0x7f79a40cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79ac53a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a4157350 con 0x7f79a40b3370 2023-11-24T06:20:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79ac53a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a40b62a0 con 0x7f79a40d1c70 2023-11-24T06:20:08.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79ac53a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a40b5690 con 0x7f79a415bdd0 2023-11-24T06:20:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aa2d6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a4156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aa2d6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a4156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37478/0 (socket says 172.21.15.89:37478) 2023-11-24T06:20:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aa2d6700 1 -- 172.21.15.89:0/2377160624 learned_addr learned my addr 172.21.15.89:0/2377160624 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79a9ad5700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40d1c70 0x7f79a40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aaad7700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 0x7f79a40cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aa2d6700 1 -- 172.21.15.89:0/2377160624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 msgr2=0x7f79a40cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aa2d6700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 0x7f79a40cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aa2d6700 1 -- 172.21.15.89:0/2377160624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40d1c70 msgr2=0x7f79a40b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aa2d6700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40d1c70 0x7f79a40b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.770+0000 7f79aa2d6700 1 -- 172.21.15.89:0/2377160624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7994009550 con 0x7f79a40b3370 2023-11-24T06:20:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.771+0000 7f79aa2d6700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a4156e10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c7f0 tx=0x7f79a000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.771+0000 7f799affd700 1 -- 172.21.15.89:0/2377160624 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0011800 con 0x7f79a40b3370 2023-11-24T06:20:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.771+0000 7f799affd700 1 -- 172.21.15.89:0/2377160624 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0011960 con 0x7f79a40b3370 2023-11-24T06:20:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.771+0000 7f799affd700 1 -- 172.21.15.89:0/2377160624 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0011cd0 con 0x7f79a40b3370 2023-11-24T06:20:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.771+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a40b5970 con 0x7f79a40b3370 2023-11-24T06:20:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.771+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a40b5f30 con 0x7f79a40b3370 2023-11-24T06:20:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.772+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a40b63e0 con 0x7f79a40b3370 2023-11-24T06:20:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.772+0000 7f799affd700 1 -- 172.21.15.89:0/2377160624 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f79a0010350 con 0x7f79a40b3370 2023-11-24T06:20:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.773+0000 7f799affd700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f798806aed0 0x7f798806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.773+0000 7f799affd700 1 -- 172.21.15.89:0/2377160624 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f79a0089140 con 0x7f79a40b3370 2023-11-24T06:20:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.773+0000 7f79a9ad5700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f798806aed0 0x7f798806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:08.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.773+0000 7f79a9ad5700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f798806aed0 0x7f798806d390 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7994009130 tx=0x7f7994008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:08.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.777+0000 7f799affd700 1 -- 172.21.15.89:0/2377160624 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79a00541a0 con 0x7f79a40b3370 2023-11-24T06:20:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.967+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79a4067f60 con 0x7f79a40b3370 2023-11-24T06:20:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.968+0000 7f799affd700 1 -- 172.21.15.89:0/2377160624 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16662 (secure 0 0 0) 0x7f79a00577c0 con 0x7f79a40b3370 2023-11-24T06:20:08.971 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:08.971 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:02.530677+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/2053513777":"2023-11-25T06:19:59.514768+0000","172.21.15.89:6835/2053513777":"2023-11-25T06:19:59.514768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:08.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f798806aed0 msgr2=0x7f798806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f798806aed0 0x7f798806d390 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7994009130 tx=0x7f7994008030 comp rx=0 tx=0).stop 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 msgr2=0x7f79a4156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a4156e10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c7f0 tx=0x7f79a000cb00 comp rx=0 tx=0).stop 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 shutdown_connections 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bdd0 0x7f79a40cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a40b3370 0x7f79a4156e10 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f798806aed0 0x7f798806d390 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 --2- 172.21.15.89:0/2377160624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40d1c70 0x7f79a40b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 >> 172.21.15.89:0/2377160624 conn(0x7f79a40731a0 msgr2=0x7f79a40b0110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 shutdown_connections 2023-11-24T06:20:08.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:08.970+0000 7f79ac53a700 1 -- 172.21.15.89:0/2377160624 wait complete. 2023-11-24T06:20:08.985 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T06:20:08.986 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:09.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc65362a700 1 Processor -- start 2023-11-24T06:20:09.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc65362a700 1 -- start start 2023-11-24T06:20:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc65362a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c15e080 0x7fc64c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc65362a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc64c0af4d0 0x7fc64c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc65362a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c0ae430 con 0x7fc64c06c000 2023-11-24T06:20:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc65362a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64c0af0b0 con 0x7fc64c0af4d0 2023-11-24T06:20:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc65362a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0b05a0 con 0x7fc64c15e080 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc650bc5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc64c0af4d0 0x7fc64c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc651bc7700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc64c06c000 0x7fc64c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40994/0 (socket says 172.21.15.89:40994) 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc651bc7700 1 -- 172.21.15.89:0/2996893859 learned_addr learned my addr 172.21.15.89:0/2996893859 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc6513c6700 1 --2- 172.21.15.89:0/2996893859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c15e080 0x7fc64c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc650bc5700 1 -- 172.21.15.89:0/2996893859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c15e080 msgr2=0x7fc64c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc650bc5700 1 --2- 172.21.15.89:0/2996893859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c15e080 0x7fc64c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc650bc5700 1 -- 172.21.15.89:0/2996893859 >> v1:172.21.15.89:6789/0 conn(0x7fc64c06c000 legacy=0x7fc64c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc650bc5700 1 -- 172.21.15.89:0/2996893859 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c0ae710 con 0x7fc64c0af4d0 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc643fff700 1 -- 172.21.15.89:0/2996893859 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214100145 0 0) 0x7fc64c0ae430 con 0x7fc64c06c000 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc6513c6700 1 --2- 172.21.15.89:0/2996893859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c15e080 0x7fc64c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.104+0000 7fc650bc5700 1 --2- 172.21.15.89:0/2996893859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c0af4d0 0x7fc64c0afb90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc63c009870 tx=0x7fc63c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b9ae17d3c4b21b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc643fff700 1 -- 172.21.15.89:0/2996893859 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63c011830 con 0x7fc64c0af4d0 2023-11-24T06:20:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc643fff700 1 -- 172.21.15.89:0/2996893859 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63c011990 con 0x7fc64c0af4d0 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc643fff700 1 -- 172.21.15.89:0/2996893859 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63c011d20 con 0x7fc64c0af4d0 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc65362a700 1 -- 172.21.15.89:0/2996893859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c0af4d0 msgr2=0x7fc64c0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc65362a700 1 --2- 172.21.15.89:0/2996893859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c0af4d0 0x7fc64c0afb90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc63c009870 tx=0x7fc63c009b80 comp rx=0 tx=0).stop 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc65362a700 1 -- 172.21.15.89:0/2996893859 shutdown_connections 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc65362a700 1 --2- 172.21.15.89:0/2996893859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c15e080 0x7fc64c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc65362a700 1 --2- 172.21.15.89:0/2996893859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c0af4d0 0x7fc64c0afb90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc65362a700 1 -- 172.21.15.89:0/2996893859 >> 172.21.15.89:0/2996893859 conn(0x7fc64c071b90 msgr2=0x7fc64c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc65362a700 1 -- 172.21.15.89:0/2996893859 shutdown_connections 2023-11-24T06:20:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.105+0000 7fc65362a700 1 -- 172.21.15.89:0/2996893859 wait complete. 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.106+0000 7fc65362a700 1 Processor -- start 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.106+0000 7fc65362a700 1 -- start start 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc65362a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc64c06c000 0x7fc64c15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc65362a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c0af4d0 0x7fc64c158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc65362a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c15e080 0x7fc64c1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc65362a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0cda80 con 0x7fc64c15e080 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc65362a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c1547e0 con 0x7fc64c06c000 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc65362a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c159850 con 0x7fc64c0af4d0 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc651bc7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c15e080 0x7fc64c1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc651bc7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c15e080 0x7fc64c1591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37496/0 (socket says 172.21.15.89:37496) 2023-11-24T06:20:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc651bc7700 1 -- 172.21.15.89:0/3012491867 learned_addr learned my addr 172.21.15.89:0/3012491867 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc6513c6700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc64c06c000 0x7fc64c15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.107+0000 7fc650bc5700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c0af4d0 0x7fc64c158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc651bc7700 1 -- 172.21.15.89:0/3012491867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c0af4d0 msgr2=0x7fc64c158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc651bc7700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c0af4d0 0x7fc64c158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc651bc7700 1 -- 172.21.15.89:0/3012491867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc64c06c000 msgr2=0x7fc64c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc651bc7700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc64c06c000 0x7fc64c15ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc651bc7700 1 -- 172.21.15.89:0/3012491867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0094a0 con 0x7fc64c15e080 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc651bc7700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c15e080 0x7fc64c1591c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc64400cca0 tx=0x7fc64400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc641ffb700 1 -- 172.21.15.89:0/3012491867 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64400ab70 con 0x7fc64c15e080 2023-11-24T06:20:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc641ffb700 1 -- 172.21.15.89:0/3012491867 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64400acd0 con 0x7fc64c15e080 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c159b30 con 0x7fc64c15e080 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc641ffb700 1 -- 172.21.15.89:0/3012491867 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644007310 con 0x7fc64c15e080 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.108+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c0cdf70 con 0x7fc64c15e080 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.109+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c0753f0 con 0x7fc64c15e080 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.109+0000 7fc641ffb700 1 -- 172.21.15.89:0/3012491867 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc644007470 con 0x7fc64c15e080 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.110+0000 7fc641ffb700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc63006aed0 0x7fc63006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.110+0000 7fc641ffb700 1 -- 172.21.15.89:0/3012491867 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc644089320 con 0x7fc64c15e080 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.110+0000 7fc6513c6700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc63006aed0 0x7fc63006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.110+0000 7fc6513c6700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc63006aed0 0x7fc63006d390 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc64c15cc40 tx=0x7fc648008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.114+0000 7fc641ffb700 1 -- 172.21.15.89:0/3012491867 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc644054400 con 0x7fc64c15e080 2023-11-24T06:20:09.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.348+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc64c0d09b0 con 0x7fc64c15e080 2023-11-24T06:20:09.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.349+0000 7fc641ffb700 1 -- 172.21.15.89:0/3012491867 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fc644057a20 con 0x7fc64c15e080 2023-11-24T06:20:09.352 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:09.352 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/503786989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":503786989},{"type":"v1","addr":"172.21.15.123:6834","nonce":503786989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3511736853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3511736853},{"type":"v1","addr":"172.21.15.89:6836","nonce":3511736853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5361,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3128631650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3128631650},{"type":"v1","addr":"172.21.15.89:6837","nonce":3128631650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:19:43.989850+0000","modified":"2023-11-24T06:19:47.380732+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6835/1601724762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1601724762},{"type":"v1","addr":"172.21.15.123:6835","nonce":1601724762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:20:09.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc63006aed0 msgr2=0x7fc63006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc63006aed0 0x7fc63006d390 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc64c15cc40 tx=0x7fc648008040 comp rx=0 tx=0).stop 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c15e080 msgr2=0x7fc64c1591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c15e080 0x7fc64c1591c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc64400cca0 tx=0x7fc64400a060 comp rx=0 tx=0).stop 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 shutdown_connections 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc64c15e080 0x7fc64c1591c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc63006aed0 0x7fc63006d390 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc64c06c000 0x7fc64c15ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 --2- 172.21.15.89:0/3012491867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc64c0af4d0 0x7fc64c158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 >> 172.21.15.89:0/3012491867 conn(0x7fc64c071b90 msgr2=0x7fc64c06e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 shutdown_connections 2023-11-24T06:20:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.352+0000 7fc65362a700 1 -- 172.21.15.89:0/3012491867 wait complete. 2023-11-24T06:20:09.356 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-11-24T06:20:09.366 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:09.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.483+0000 7f54b8e78700 1 Processor -- start 2023-11-24T06:20:09.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.483+0000 7f54b8e78700 1 -- start start 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b8e78700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 0x7f54b41604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b8e78700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54b40b3370 0x7f54b40b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b8e78700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b40b2030 con 0x7f54b40d1c70 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b8e78700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b40b2f50 con 0x7f54b40b3370 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b8e78700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40b22d0 con 0x7f54b415bdd0 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b1d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54b40b3370 0x7f54b40b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b2d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f54b40d1c70 0x7f54b40ce6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41008/0 (socket says 172.21.15.89:41008) 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b2d9d700 1 -- 172.21.15.89:0/2085461482 learned_addr learned my addr 172.21.15.89:0/2085461482 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:09.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.484+0000 7f54b259c700 1 --2- 172.21.15.89:0/2085461482 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 0x7f54b41604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b1d9b700 1 -- 172.21.15.89:0/2085461482 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 msgr2=0x7f54b41604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b1d9b700 1 --2- 172.21.15.89:0/2085461482 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 0x7f54b41604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b1d9b700 1 -- 172.21.15.89:0/2085461482 >> v1:172.21.15.89:6789/0 conn(0x7f54b40d1c70 legacy=0x7f54b40ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b1d9b700 1 -- 172.21.15.89:0/2085461482 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b40b25b0 con 0x7f54b40b3370 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b159a700 1 -- 172.21.15.89:0/2085461482 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147578541 0 0) 0x7f54b40b2030 con 0x7f54b40d1c70 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b1d9b700 1 --2- 172.21.15.89:0/2085461482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40b3370 0x7f54b40b1af0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009840 tx=0x7f54a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5c904fdc9e0730 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b159a700 1 -- 172.21.15.89:0/2085461482 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4011830 con 0x7f54b40b3370 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b159a700 1 -- 172.21.15.89:0/2085461482 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4011990 con 0x7f54b40b3370 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b159a700 1 -- 172.21.15.89:0/2085461482 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4011d20 con 0x7f54b40b3370 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b8e78700 1 -- 172.21.15.89:0/2085461482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40b3370 msgr2=0x7f54b40b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.485+0000 7f54b8e78700 1 --2- 172.21.15.89:0/2085461482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40b3370 0x7f54b40b1af0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009840 tx=0x7f54a4009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:09.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.486+0000 7f54b8e78700 1 -- 172.21.15.89:0/2085461482 shutdown_connections 2023-11-24T06:20:09.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.486+0000 7f54b8e78700 1 --2- 172.21.15.89:0/2085461482 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 0x7f54b41604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.486+0000 7f54b8e78700 1 --2- 172.21.15.89:0/2085461482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40b3370 0x7f54b40b1af0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.486+0000 7f54b8e78700 1 -- 172.21.15.89:0/2085461482 >> 172.21.15.89:0/2085461482 conn(0x7f54b40731a0 msgr2=0x7f54b409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:09.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.486+0000 7f54b8e78700 1 -- 172.21.15.89:0/2085461482 shutdown_connections 2023-11-24T06:20:09.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.486+0000 7f54b8e78700 1 -- 172.21.15.89:0/2085461482 wait complete. 2023-11-24T06:20:09.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.487+0000 7f54b8e78700 1 Processor -- start 2023-11-24T06:20:09.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.487+0000 7f54b8e78700 1 -- start start 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b8e78700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b40b3370 0x7f54b414fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b8e78700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40d1c70 0x7f54b415c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b8e78700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 0x7f54b415c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b8e78700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41503e0 con 0x7f54b40d1c70 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b8e78700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40b51e0 con 0x7f54b40b3370 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b8e78700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40b48d0 con 0x7f54b415bdd0 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40d1c70 0x7f54b415c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40d1c70 0x7f54b415c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37508/0 (socket says 172.21.15.89:37508) 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 -- 172.21.15.89:0/1498646493 learned_addr learned my addr 172.21.15.89:0/1498646493 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:09.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b259c700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b40b3370 0x7f54b414fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 -- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 msgr2=0x7f54b415c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b2d9d700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 0x7f54b415c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 0x7f54b415c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 -- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b40b3370 msgr2=0x7f54b414fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b40b3370 0x7f54b414fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 -- 172.21.15.89:0/1498646493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a40094a0 con 0x7f54b40d1c70 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b259c700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b40b3370 0x7f54b414fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b1d9b700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40d1c70 0x7f54b415c4b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009fd0 tx=0x7f54a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54a2ffd700 1 -- 172.21.15.89:0/1498646493 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a40038a0 con 0x7f54b40d1c70 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54a2ffd700 1 -- 172.21.15.89:0/1498646493 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a401f070 con 0x7f54b40d1c70 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b40cb640 con 0x7f54b40d1c70 2023-11-24T06:20:09.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.488+0000 7f54a2ffd700 1 -- 172.21.15.89:0/1498646493 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4003c40 con 0x7f54b40d1c70 2023-11-24T06:20:09.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.489+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b40b4e70 con 0x7f54b40d1c70 2023-11-24T06:20:09.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.489+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b4067f60 con 0x7f54b40d1c70 2023-11-24T06:20:09.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.490+0000 7f54a2ffd700 1 -- 172.21.15.89:0/1498646493 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f54a4026480 con 0x7f54b40d1c70 2023-11-24T06:20:09.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.490+0000 7f54a2ffd700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f549406aed0 0x7f549406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.490+0000 7f54a2ffd700 1 -- 172.21.15.89:0/1498646493 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f54a4021070 con 0x7f54b40d1c70 2023-11-24T06:20:09.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.490+0000 7f54b259c700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f549406aed0 0x7f549406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.491+0000 7f54b259c700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f549406aed0 0x7f549406d390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009ac0 tx=0x7f54a8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.495+0000 7f54a2ffd700 1 -- 172.21.15.89:0/1498646493 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54a4056030 con 0x7f54b40d1c70 2023-11-24T06:20:09.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.726+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54b40b5000 con 0x7f54b40d1c70 2023-11-24T06:20:09.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.727+0000 7f54a2ffd700 1 -- 172.21.15.89:0/1498646493 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f54a4015090 con 0x7f54b40d1c70 2023-11-24T06:20:09.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:09.731 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/503786989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":503786989},{"type":"v1","addr":"172.21.15.123:6834","nonce":503786989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3511736853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3511736853},{"type":"v1","addr":"172.21.15.89:6836","nonce":3511736853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5361,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3128631650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3128631650},{"type":"v1","addr":"172.21.15.89:6837","nonce":3128631650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:19:43.989850+0000","modified":"2023-11-24T06:19:47.380732+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6835/1601724762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1601724762},{"type":"v1","addr":"172.21.15.123:6835","nonce":1601724762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:20:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f549406aed0 msgr2=0x7f549406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f549406aed0 0x7f549406d390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009ac0 tx=0x7f54a8008040 comp rx=0 tx=0).stop 2023-11-24T06:20:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40d1c70 msgr2=0x7f54b415c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40d1c70 0x7f54b415c4b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009fd0 tx=0x7f54a4004750 comp rx=0 tx=0).stop 2023-11-24T06:20:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 shutdown_connections 2023-11-24T06:20:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54b415bdd0 0x7f54b415c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f549406aed0 0x7f549406d390 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54b40b3370 0x7f54b414fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 --2- 172.21.15.89:0/1498646493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54b40d1c70 0x7f54b415c4b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 >> 172.21.15.89:0/1498646493 conn(0x7f54b40731a0 msgr2=0x7f54b40afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 shutdown_connections 2023-11-24T06:20:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.730+0000 7f54b8e78700 1 -- 172.21.15.89:0/1498646493 wait complete. 2023-11-24T06:20:09.733 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 22 2023-11-24T06:20:09.744 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:09.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.862+0000 7f0bae865700 1 Processor -- start 2023-11-24T06:20:09.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.862+0000 7f0bae865700 1 -- start start 2023-11-24T06:20:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0bae865700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80b3370 0x7f0ba81604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0bae865700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ba80d1c70 0x7f0ba80d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0bae865700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80b22d0 con 0x7f0ba815bdd0 2023-11-24T06:20:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0bae865700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba80b2f50 con 0x7f0ba80d1c70 2023-11-24T06:20:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0bae865700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80d2640 con 0x7f0ba80b3370 2023-11-24T06:20:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0ba7fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0ba815bdd0 0x7f0ba80b1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41018/0 (socket says 172.21.15.89:41018) 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0ba7fff700 1 -- 172.21.15.89:0/3675522582 learned_addr learned my addr 172.21.15.89:0/3675522582 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0ba77fe700 1 --2- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80b3370 0x7f0ba81604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0bace02700 1 --2- 172.21.15.89:0/3675522582 >> v2:172.21.15.123:3300/0 conn(0x7f0ba80d1c70 0x7f0ba80d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.863+0000 7f0ba6ffd700 1 -- 172.21.15.89:0/3675522582 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291218559 0 0) 0x7f0ba80b22d0 con 0x7f0ba815bdd0 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0ba6ffd700 1 -- 172.21.15.89:0/3675522582 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b900035b0 con 0x7f0ba815bdd0 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0bace02700 1 -- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80b3370 msgr2=0x7f0ba81604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0bace02700 1 --2- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80b3370 0x7f0ba81604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0bace02700 1 -- 172.21.15.89:0/3675522582 >> v1:172.21.15.89:6789/0 conn(0x7f0ba815bdd0 legacy=0x7f0ba80b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0bace02700 1 -- 172.21.15.89:0/3675522582 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80b25b0 con 0x7f0ba80d1c70 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0ba77fe700 1 --2- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80b3370 0x7f0ba81604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0bace02700 1 --2- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba80d1c70 0x7f0ba80d2100 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009d90 tx=0x7f0ba000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49a63de6dbbc6beb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0ba6ffd700 1 -- 172.21.15.89:0/3675522582 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba000e070 con 0x7f0ba80d1c70 2023-11-24T06:20:09.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0ba6ffd700 1 -- 172.21.15.89:0/3675522582 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba000bb20 con 0x7f0ba80d1c70 2023-11-24T06:20:09.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.864+0000 7f0ba6ffd700 1 -- 172.21.15.89:0/3675522582 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba000be90 con 0x7f0ba80d1c70 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.865+0000 7f0bae865700 1 -- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba80d1c70 msgr2=0x7f0ba80d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.865+0000 7f0bae865700 1 --2- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba80d1c70 0x7f0ba80d2100 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009d90 tx=0x7f0ba000b330 comp rx=0 tx=0).stop 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.865+0000 7f0bae865700 1 -- 172.21.15.89:0/3675522582 shutdown_connections 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.865+0000 7f0bae865700 1 --2- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba80d1c70 0x7f0ba80d2100 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.865+0000 7f0bae865700 1 --2- 172.21.15.89:0/3675522582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80b3370 0x7f0ba81604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.865+0000 7f0bae865700 1 -- 172.21.15.89:0/3675522582 >> 172.21.15.89:0/3675522582 conn(0x7f0ba80731a0 msgr2=0x7f0ba809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.865+0000 7f0bae865700 1 -- 172.21.15.89:0/3675522582 shutdown_connections 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.865+0000 7f0bae865700 1 -- 172.21.15.89:0/3675522582 wait complete. 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.866+0000 7f0bae865700 1 Processor -- start 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.866+0000 7f0bae865700 1 -- start start 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0bae865700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba80b3370 0x7f0ba80cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0bae865700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80d1c70 0x7f0ba80cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0bae865700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba815bdd0 0x7f0ba815d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0bae865700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80ca450 con 0x7f0ba80b3370 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0bae865700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80ca6f0 con 0x7f0ba815bdd0 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0bae865700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80ca9d0 con 0x7f0ba80d1c70 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba80b3370 0x7f0ba80cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba80b3370 0x7f0ba80cdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37522/0 (socket says 172.21.15.89:37522) 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1994056246 learned_addr learned my addr 172.21.15.89:0/1994056246 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba77fe700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80d1c70 0x7f0ba80cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0bace02700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba815bdd0 0x7f0ba815d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80d1c70 msgr2=0x7f0ba80cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80d1c70 0x7f0ba80cb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba815bdd0 msgr2=0x7f0ba815d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba815bdd0 0x7f0ba815d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 -- 172.21.15.89:0/1994056246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c0094a0 con 0x7f0ba80b3370 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0bace02700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba815bdd0 0x7f0ba815d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.867+0000 7f0ba7fff700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba80b3370 0x7f0ba80cdd80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c420 tx=0x7f0b9800c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.868+0000 7f0ba4ff9700 1 -- 172.21.15.89:0/1994056246 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98005d70 con 0x7f0ba80b3370 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.868+0000 7f0ba4ff9700 1 -- 172.21.15.89:0/1994056246 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b980070d0 con 0x7f0ba80b3370 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.868+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba0009a70 con 0x7f0ba80b3370 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.868+0000 7f0ba4ff9700 1 -- 172.21.15.89:0/1994056246 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98007440 con 0x7f0ba80b3370 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.868+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba80caf90 con 0x7f0ba80b3370 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.868+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba80cb150 con 0x7f0ba80b3370 2023-11-24T06:20:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.869+0000 7f0ba4ff9700 1 -- 172.21.15.89:0/1994056246 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0b980075a0 con 0x7f0ba80b3370 2023-11-24T06:20:09.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.869+0000 7f0ba4ff9700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0b9406aed0 0x7f0b9406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:09.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.869+0000 7f0ba4ff9700 1 -- 172.21.15.89:0/1994056246 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0b980888e0 con 0x7f0ba80b3370 2023-11-24T06:20:09.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.869+0000 7f0ba77fe700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0b9406aed0 0x7f0b9406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:09.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.870+0000 7f0ba77fe700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0b9406aed0 0x7f0b9406d390 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80ca2e0 tx=0x7f0b9c009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:09.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:09.874+0000 7f0ba4ff9700 1 -- 172.21.15.89:0/1994056246 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b980539e0 con 0x7f0ba80b3370 2023-11-24T06:20:10.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.066+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ba8160180 con 0x7f0ba80b3370 2023-11-24T06:20:10.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.067+0000 7f0ba4ff9700 1 -- 172.21.15.89:0/1994056246 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16662 (secure 0 0 0) 0x7f0b98017020 con 0x7f0ba80b3370 2023-11-24T06:20:10.069 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:10.069 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:02.530677+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:19:41.783017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:19:42.953803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/2053513777":"2023-11-25T06:19:59.514768+0000","172.21.15.89:6835/2053513777":"2023-11-25T06:19:59.514768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:10.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0b9406aed0 msgr2=0x7f0b9406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0b9406aed0 0x7f0b9406d390 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80ca2e0 tx=0x7f0b9c009210 comp rx=0 tx=0).stop 2023-11-24T06:20:10.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba80b3370 msgr2=0x7f0ba80cdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba80b3370 0x7f0ba80cdd80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c420 tx=0x7f0b9800c730 comp rx=0 tx=0).stop 2023-11-24T06:20:10.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 shutdown_connections 2023-11-24T06:20:10.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0b9406aed0 0x7f0b9406d390 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba815bdd0 0x7f0ba815d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba80b3370 0x7f0ba80cdd80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 --2- 172.21.15.89:0/1994056246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ba80d1c70 0x7f0ba80cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 >> 172.21.15.89:0/1994056246 conn(0x7f0ba80731a0 msgr2=0x7f0ba809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 shutdown_connections 2023-11-24T06:20:10.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.069+0000 7f0bae865700 1 -- 172.21.15.89:0/1994056246 wait complete. 2023-11-24T06:20:10.084 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T06:20:10.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.203+0000 7fd17ece6700 1 Processor -- start 2023-11-24T06:20:10.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.203+0000 7fd17ece6700 1 -- start start 2023-11-24T06:20:10.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd17ece6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 0x7fd1780af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd17ece6700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1780b0f80 0x7fd1780cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd17ece6700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780b2bd0 con 0x7fd17815ee90 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd17ece6700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1780b1fc0 con 0x7fd1780b0f80 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd17ece6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1780b22a0 con 0x7fd1780b0ba0 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd177fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1780b0f80 0x7fd1780cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd17d283700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd17815ee90 0x7fd1780ccca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41036/0 (socket says 172.21.15.89:41036) 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd17d283700 1 -- 172.21.15.89:0/1129797679 learned_addr learned my addr 172.21.15.89:0/1129797679 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd17ca82700 1 --2- 172.21.15.89:0/1129797679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 0x7fd1780af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd177fff700 1 -- 172.21.15.89:0/1129797679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 msgr2=0x7fd1780af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd177fff700 1 --2- 172.21.15.89:0/1129797679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 0x7fd1780af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd177fff700 1 -- 172.21.15.89:0/1129797679 >> v1:172.21.15.89:6789/0 conn(0x7fd17815ee90 legacy=0x7fd1780ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd177fff700 1 -- 172.21.15.89:0/1129797679 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1780b2580 con 0x7fd1780b0f80 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd1777fe700 1 -- 172.21.15.89:0/1129797679 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936981147 0 0) 0x7fd1780b2bd0 con 0x7fd17815ee90 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd177fff700 1 --2- 172.21.15.89:0/1129797679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1780b0f80 0x7fd1780cc760 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd164009840 tx=0x7fd164009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f9e5fc46ed170b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:10.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.204+0000 7fd1777fe700 1 -- 172.21.15.89:0/1129797679 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164011830 con 0x7fd1780b0f80 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd1777fe700 1 -- 172.21.15.89:0/1129797679 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd164011990 con 0x7fd1780b0f80 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd1777fe700 1 -- 172.21.15.89:0/1129797679 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164011d20 con 0x7fd1780b0f80 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd17ece6700 1 -- 172.21.15.89:0/1129797679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1780b0f80 msgr2=0x7fd1780cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1129797679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1780b0f80 0x7fd1780cc760 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd164009840 tx=0x7fd164009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd17ece6700 1 -- 172.21.15.89:0/1129797679 shutdown_connections 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1129797679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 0x7fd1780af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1129797679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1780b0f80 0x7fd1780cc760 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd17ece6700 1 -- 172.21.15.89:0/1129797679 >> 172.21.15.89:0/1129797679 conn(0x7fd1780b57b0 msgr2=0x7fd1780b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd17ece6700 1 -- 172.21.15.89:0/1129797679 shutdown_connections 2023-11-24T06:20:10.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.205+0000 7fd17ece6700 1 -- 172.21.15.89:0/1129797679 wait complete. 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.206+0000 7fd17ece6700 1 Processor -- start 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.206+0000 7fd17ece6700 1 -- start start 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17ece6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 0x7fd17814e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17ece6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1780b0f80 0x7fd1781504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17ece6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd17815ee90 0x7fd17814a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17ece6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1780b4940 con 0x7fd17815ee90 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17ece6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1780b5150 con 0x7fd1780b0f80 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17ece6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd178159ec0 con 0x7fd1780b0ba0 2023-11-24T06:20:10.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17d283700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd17815ee90 0x7fd17814a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17d283700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd17815ee90 0x7fd17814a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37538/0 (socket says 172.21.15.89:37538) 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17d283700 1 -- 172.21.15.89:0/1551264437 learned_addr learned my addr 172.21.15.89:0/1551264437 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd17ca82700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 0x7fd17814e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.207+0000 7fd177fff700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1780b0f80 0x7fd1781504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd17d283700 1 -- 172.21.15.89:0/1551264437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 msgr2=0x7fd17814e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd17d283700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 0x7fd17814e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd17d283700 1 -- 172.21.15.89:0/1551264437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1780b0f80 msgr2=0x7fd1781504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd17d283700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1780b0f80 0x7fd1781504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd17d283700 1 -- 172.21.15.89:0/1551264437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640094a0 con 0x7fd17815ee90 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd17d283700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd17815ee90 0x7fd17814a3b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd17000cba0 tx=0x7fd17000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd1757fa700 1 -- 172.21.15.89:0/1551264437 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17000ab80 con 0x7fd17815ee90 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd1757fa700 1 -- 172.21.15.89:0/1551264437 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17000ad00 con 0x7fd17815ee90 2023-11-24T06:20:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd1757fa700 1 -- 172.21.15.89:0/1551264437 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170007310 con 0x7fd17815ee90 2023-11-24T06:20:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17814ee20 con 0x7fd17815ee90 2023-11-24T06:20:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.208+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1780b4e70 con 0x7fd17815ee90 2023-11-24T06:20:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.209+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd178067f60 con 0x7fd17815ee90 2023-11-24T06:20:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.209+0000 7fd1757fa700 1 -- 172.21.15.89:0/1551264437 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd170007470 con 0x7fd17815ee90 2023-11-24T06:20:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.210+0000 7fd1757fa700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd15c06aed0 0x7fd15c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.210+0000 7fd1757fa700 1 -- 172.21.15.89:0/1551264437 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd170089090 con 0x7fd17815ee90 2023-11-24T06:20:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.210+0000 7fd17ca82700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd15c06aed0 0x7fd15c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.210+0000 7fd17ca82700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd15c06aed0 0x7fd15c06d390 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009930 tx=0x7fd16c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.214+0000 7fd1757fa700 1 -- 172.21.15.89:0/1551264437 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd170054170 con 0x7fd17815ee90 2023-11-24T06:20:10.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.447+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd17814a0d0 con 0x7fd17815ee90 2023-11-24T06:20:10.449 INFO:tasks.ceph.mon.a.smithi089.stderr:2023-11-24T06:20:10.448+0000 7f7eccc4c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T06:20:10.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.449+0000 7fd1757fa700 1 -- 172.21.15.89:0/1551264437 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd17814a0d0 con 0x7fd17815ee90 2023-11-24T06:20:10.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd15c06aed0 msgr2=0x7fd15c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd15c06aed0 0x7fd15c06d390 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009930 tx=0x7fd16c008040 comp rx=0 tx=0).stop 2023-11-24T06:20:10.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd17815ee90 msgr2=0x7fd17814a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd17815ee90 0x7fd17814a3b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd17000cba0 tx=0x7fd17000cf60 comp rx=0 tx=0).stop 2023-11-24T06:20:10.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 shutdown_connections 2023-11-24T06:20:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd17815ee90 0x7fd17814a3b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1780b0ba0 0x7fd17814e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd15c06aed0 0x7fd15c06d390 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 --2- 172.21.15.89:0/1551264437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1780b0f80 0x7fd1781504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 >> 172.21.15.89:0/1551264437 conn(0x7fd1780b57b0 msgr2=0x7fd178154210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 shutdown_connections 2023-11-24T06:20:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.451+0000 7fd17ece6700 1 -- 172.21.15.89:0/1551264437 wait complete. 2023-11-24T06:20:10.456 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T06:20:10.457 INFO:tasks.ceph.mds.b.smithi123.stderr: -3629> 2023-11-24T06:19:48.472+0000 7ff35c4d8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:20:10.467 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T06:20:10.490 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2023-11-24T06:20:10.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.590+0000 7fad36424700 1 Processor -- start 2023-11-24T06:20:10.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.590+0000 7fad36424700 1 -- start start 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad36424700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fad3015e160 0x7fad300b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad36424700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 0x7fad30073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad36424700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fad300744d0 0x7fad3006d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad36424700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad300b4500 con 0x7fad300744d0 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad36424700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad30074960 con 0x7fad3015e160 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad36424700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad300b3c60 con 0x7fad300afc90 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad349c1700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fad300744d0 0x7fad3006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad349c1700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fad300744d0 0x7fad3006d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37546/0 (socket says 172.21.15.89:37546) 2023-11-24T06:20:10.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.591+0000 7fad349c1700 1 -- 172.21.15.89:0/1312839596 learned_addr learned my addr 172.21.15.89:0/1312839596 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad2f7fe700 1 --2- 172.21.15.89:0/1312839596 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 0x7fad30073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad2ffff700 1 --2- 172.21.15.89:0/1312839596 >> v2:172.21.15.123:3300/0 conn(0x7fad3015e160 0x7fad300b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad349c1700 1 -- 172.21.15.89:0/1312839596 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 msgr2=0x7fad30073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad349c1700 1 --2- 172.21.15.89:0/1312839596 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 0x7fad30073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad349c1700 1 -- 172.21.15.89:0/1312839596 >> v2:172.21.15.123:3300/0 conn(0x7fad3015e160 msgr2=0x7fad300b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad349c1700 1 --2- 172.21.15.89:0/1312839596 >> v2:172.21.15.123:3300/0 conn(0x7fad3015e160 0x7fad300b5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad349c1700 1 -- 172.21.15.89:0/1312839596 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad300b58c0 con 0x7fad300744d0 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad349c1700 1 --2- 172.21.15.89:0/1312839596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad3006d930 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fad3009efa0 tx=0x7fad2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f7f9af6a76c79e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad2effd700 1 -- 172.21.15.89:0/1312839596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2800e070 con 0x7fad300744d0 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad2effd700 1 -- 172.21.15.89:0/1312839596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2800b9c0 con 0x7fad300744d0 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad2effd700 1 -- 172.21.15.89:0/1312839596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2800bd70 con 0x7fad300744d0 2023-11-24T06:20:10.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad36424700 1 -- 172.21.15.89:0/1312839596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 msgr2=0x7fad3006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad36424700 1 --2- 172.21.15.89:0/1312839596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad3006d930 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fad3009efa0 tx=0x7fad2800b330 comp rx=0 tx=0).stop 2023-11-24T06:20:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.592+0000 7fad36424700 1 -- 172.21.15.89:0/1312839596 shutdown_connections 2023-11-24T06:20:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.593+0000 7fad36424700 1 --2- 172.21.15.89:0/1312839596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad3006d930 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.593+0000 7fad36424700 1 --2- 172.21.15.89:0/1312839596 >> v2:172.21.15.123:3300/0 conn(0x7fad3015e160 0x7fad300b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.593+0000 7fad36424700 1 --2- 172.21.15.89:0/1312839596 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 0x7fad30073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.593+0000 7fad36424700 1 -- 172.21.15.89:0/1312839596 >> 172.21.15.89:0/1312839596 conn(0x7fad300d0bf0 msgr2=0x7fad300d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.593+0000 7fad36424700 1 -- 172.21.15.89:0/1312839596 shutdown_connections 2023-11-24T06:20:10.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.593+0000 7fad36424700 1 -- 172.21.15.89:0/1312839596 wait complete. 2023-11-24T06:20:10.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.594+0000 7fad36424700 1 Processor -- start 2023-11-24T06:20:10.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.594+0000 7fad36424700 1 -- start start 2023-11-24T06:20:10.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad36424700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad301604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad36424700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 0x7fad30151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad36424700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad3015e160 0x7fad30151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad36424700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad30153790 con 0x7fad300744d0 2023-11-24T06:20:10.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad36424700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad300b2680 con 0x7fad3015e160 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad36424700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad301f6f20 con 0x7fad300afc90 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad301604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad301604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37548/0 (socket says 172.21.15.89:37548) 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 -- 172.21.15.89:0/890704529 learned_addr learned my addr 172.21.15.89:0/890704529 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2f7fe700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 0x7fad30151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad349c1700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad3015e160 0x7fad30151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 -- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 msgr2=0x7fad30151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 0x7fad30151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 -- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad3015e160 msgr2=0x7fad30151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad3015e160 0x7fad30151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 -- 172.21.15.89:0/890704529 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad28009b30 con 0x7fad300744d0 2023-11-24T06:20:10.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad349c1700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad3015e160 0x7fad30151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2ffff700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad301604e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fad2400c7f0 tx=0x7fad2400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2cff9700 1 -- 172.21.15.89:0/890704529 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24011800 con 0x7fad300744d0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2cff9700 1 -- 172.21.15.89:0/890704529 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad24011960 con 0x7fad300744d0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad301f7200 con 0x7fad300744d0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.595+0000 7fad2cff9700 1 -- 172.21.15.89:0/890704529 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24011cf0 con 0x7fad300744d0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.596+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad301f77c0 con 0x7fad300744d0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.596+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad301f7950 con 0x7fad300744d0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.597+0000 7fad2cff9700 1 -- 172.21.15.89:0/890704529 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fad24018360 con 0x7fad300744d0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.597+0000 7fad2cff9700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fad1c06aed0 0x7fad1c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.597+0000 7fad2cff9700 1 -- 172.21.15.89:0/890704529 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fad240895b0 con 0x7fad300744d0 2023-11-24T06:20:10.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.597+0000 7fad2f7fe700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fad1c06aed0 0x7fad1c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:10.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.598+0000 7fad2f7fe700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fad1c06aed0 0x7fad1c06d390 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fad3006c530 tx=0x7fad20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:10.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.602+0000 7fad2cff9700 1 -- 172.21.15.89:0/890704529 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad24054510 con 0x7fad300744d0 2023-11-24T06:20:10.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:10.836+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fad30068fb0 con 0x7fad300744d0 2023-11-24T06:20:11.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.450+0000 7fad2cff9700 1 -- 172.21.15.89:0/890704529 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7fad24057b30 con 0x7fad300744d0 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fad1c06aed0 msgr2=0x7fad1c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fad1c06aed0 0x7fad1c06d390 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fad3006c530 tx=0x7fad20008040 comp rx=0 tx=0).stop 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 msgr2=0x7fad301604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad301604e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fad2400c7f0 tx=0x7fad2400cbb0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 shutdown_connections 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fad1c06aed0 0x7fad1c06d390 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad3015e160 0x7fad30151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad300744d0 0x7fad301604e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 --2- 172.21.15.89:0/890704529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad300afc90 0x7fad30151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 >> 172.21.15.89:0/890704529 conn(0x7fad300d0bf0 msgr2=0x7fad3006e260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:11.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 shutdown_connections 2023-11-24T06:20:11.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.453+0000 7fad36424700 1 -- 172.21.15.89:0/890704529 wait complete. 2023-11-24T06:20:11.470 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T06:20:11.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.599+0000 7f5491d50700 1 Processor -- start 2023-11-24T06:20:11.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.599+0000 7f5491d50700 1 -- start start 2023-11-24T06:20:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f5491d50700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 0x7f548c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f5491d50700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f548c06e830 0x7f548c06eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f5491d50700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c06c9b0 con 0x7f548c06f480 2023-11-24T06:20:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f5491d50700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548c06aff0 con 0x7f548c06e830 2023-11-24T06:20:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f5491d50700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c06f1e0 con 0x7f548c144600 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f548bfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f548c06e830 0x7f548c06eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f548bfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f548c06e830 0x7f548c06eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37560/0 (socket says 172.21.15.89:37560) 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f548bfff700 1 -- 172.21.15.89:0/930763858 learned_addr learned my addr 172.21.15.89:0/930763858 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f548bfff700 1 -- 172.21.15.89:0/930763858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 msgr2=0x7f548c0b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f548bfff700 1 --2- 172.21.15.89:0/930763858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 0x7f548c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f548bfff700 1 -- 172.21.15.89:0/930763858 >> v1:172.21.15.123:6789/0 conn(0x7f548c06f480 legacy=0x7f548c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.600+0000 7f548bfff700 1 -- 172.21.15.89:0/930763858 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c06b200 con 0x7f548c06e830 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f548bfff700 1 --2- 172.21.15.89:0/930763858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06e830 0x7f548c06eca0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5484009e50 tx=0x7f548400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1387ed5237445c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f548a7fc700 1 -- 172.21.15.89:0/930763858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484003c20 con 0x7f548c06e830 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f548a7fc700 1 -- 172.21.15.89:0/930763858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5484003d80 con 0x7f548c06e830 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f548a7fc700 1 -- 172.21.15.89:0/930763858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548400bbc0 con 0x7f548c06e830 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f5491d50700 1 -- 172.21.15.89:0/930763858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06e830 msgr2=0x7f548c06eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f5491d50700 1 --2- 172.21.15.89:0/930763858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06e830 0x7f548c06eca0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5484009e50 tx=0x7f548400b330 comp rx=0 tx=0).stop 2023-11-24T06:20:11.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f5491d50700 1 -- 172.21.15.89:0/930763858 shutdown_connections 2023-11-24T06:20:11.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f5491d50700 1 --2- 172.21.15.89:0/930763858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06e830 0x7f548c06eca0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f5491d50700 1 --2- 172.21.15.89:0/930763858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 0x7f548c0b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f5491d50700 1 -- 172.21.15.89:0/930763858 >> 172.21.15.89:0/930763858 conn(0x7f548c0d0b40 msgr2=0x7f548c0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:11.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f5491d50700 1 -- 172.21.15.89:0/930763858 shutdown_connections 2023-11-24T06:20:11.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.601+0000 7f5491d50700 1 -- 172.21.15.89:0/930763858 wait complete. 2023-11-24T06:20:11.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.602+0000 7f5491d50700 1 Processor -- start 2023-11-24T06:20:11.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f5491d50700 1 -- start start 2023-11-24T06:20:11.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f5491d50700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f548c06e830 0x7f548c0ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:11.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f5491d50700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06f480 0x7f548c0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f5491d50700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 0x7f548c0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f5491d50700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c15e1a0 con 0x7f548c06f480 2023-11-24T06:20:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f5491d50700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c15d100 con 0x7f548c06e830 2023-11-24T06:20:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f5491d50700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c15d3e0 con 0x7f548c144600 2023-11-24T06:20:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f548affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06f480 0x7f548c0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f548affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06f480 0x7f548c0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37564/0 (socket says 172.21.15.89:37564) 2023-11-24T06:20:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.603+0000 7f548affd700 1 -- 172.21.15.89:0/2390469571 learned_addr learned my addr 172.21.15.89:0/2390469571 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f548b7fe700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f548c06e830 0x7f548c0ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f548bfff700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 0x7f548c0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f548affd700 1 -- 172.21.15.89:0/2390469571 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 msgr2=0x7f548c0af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f548affd700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 0x7f548c0af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f548affd700 1 -- 172.21.15.89:0/2390469571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f548c06e830 msgr2=0x7f548c0ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f548affd700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f548c06e830 0x7f548c0ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f548affd700 1 -- 172.21.15.89:0/2390469571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484009b30 con 0x7f548c06f480 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f548affd700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06f480 0x7f548c0ceaa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f547c00c4a0 tx=0x7f547c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f5473fff700 1 -- 172.21.15.89:0/2390469571 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c0054e0 con 0x7f548c06f480 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f5473fff700 1 -- 172.21.15.89:0/2390469571 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c003680 con 0x7f548c06f480 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f5473fff700 1 -- 172.21.15.89:0/2390469571 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c003a30 con 0x7f548c06f480 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c15d6c0 con 0x7f548c06f480 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.604+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c15dc80 con 0x7f548c06f480 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.605+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c15de40 con 0x7f548c06f480 2023-11-24T06:20:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.605+0000 7f5473fff700 1 -- 172.21.15.89:0/2390469571 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f547c003b90 con 0x7f548c06f480 2023-11-24T06:20:11.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.606+0000 7f5473fff700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f547806aed0 0x7f547806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:11.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.606+0000 7f5473fff700 1 -- 172.21.15.89:0/2390469571 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f547c088540 con 0x7f548c06f480 2023-11-24T06:20:11.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.606+0000 7f548b7fe700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f547806aed0 0x7f547806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:11.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.606+0000 7f548b7fe700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f547806aed0 0x7f547806d390 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5480009970 tx=0x7f5480009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:11.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.611+0000 7f5473fff700 1 -- 172.21.15.89:0/2390469571 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f547c0535d0 con 0x7f548c06f480 2023-11-24T06:20:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:11.803+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f548c067f60 con 0x7f548c06f480 2023-11-24T06:20:12.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.571+0000 7f5473fff700 1 -- 172.21.15.89:0/2390469571 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) v1 ==== 181+0+0 (secure 0 0 0) 0x7f547c056bf0 con 0x7f548c06f480 2023-11-24T06:20:12.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f547806aed0 msgr2=0x7f547806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:12.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f547806aed0 0x7f547806d390 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5480009970 tx=0x7f5480009110 comp rx=0 tx=0).stop 2023-11-24T06:20:12.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06f480 msgr2=0x7f548c0ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06f480 0x7f548c0ceaa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f547c00c4a0 tx=0x7f547c00c7b0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 shutdown_connections 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f548c144600 0x7f548c0af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f547806aed0 0x7f547806d390 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f548c06e830 0x7f548c0ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 --2- 172.21.15.89:0/2390469571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c06f480 0x7f548c0ceaa0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 >> 172.21.15.89:0/2390469571 conn(0x7f548c0d0b40 msgr2=0x7f548c1438d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 shutdown_connections 2023-11-24T06:20:12.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.575+0000 7f5491d50700 1 -- 172.21.15.89:0/2390469571 wait complete. 2023-11-24T06:20:12.579 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2023-11-24T06:20:12.592 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T06:20:12.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.721+0000 7f5324237700 1 Processor -- start 2023-11-24T06:20:12.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.721+0000 7f5324237700 1 -- start start 2023-11-24T06:20:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5324237700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c144600 0x7f531c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5324237700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c06c9b0 con 0x7f531c06f480 2023-11-24T06:20:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5324237700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c06b200 con 0x7f531c06e830 2023-11-24T06:20:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5324237700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c06aff0 con 0x7f531c144600 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f53217d2700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f531c06f480 0x7f531c06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41074/0 (socket says 172.21.15.89:41074) 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f53217d2700 1 -- 172.21.15.89:0/1700145728 learned_addr learned my addr 172.21.15.89:0/1700145728 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924704407 0 0) 0x7f531c06c9b0 con 0x7f531c06f480 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53080035b0 con 0x7f531c06f480 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5321fd3700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c144600 0x7f531c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3668526473 0 0) 0x7f53080035b0 con 0x7f531c06f480 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c144600 msgr2=0x7f531c0b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c144600 0x7f531c0b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 >> v1:172.21.15.123:6789/0 conn(0x7f531c06e830 legacy=0x7f531c0cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c06b4e0 con 0x7f531c06f480 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f530c001670 con 0x7f531c06f480 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 >> v1:172.21.15.89:6789/0 conn(0x7f531c06f480 legacy=0x7f531c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5308003a30 0x7f5308003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308004850 0x7f5308004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c144600 0x7f5308005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c06aff0 con 0x7f5308003a30 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530800bb40 con 0x7f5308004850 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530800bd00 con 0x7f531c144600 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.722+0000 7f5321fd3700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5308003a30 0x7f5308003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f53217d2700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308004850 0x7f5308004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f53227d4700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c144600 0x7f5308005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5321fd3700 1 -- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c144600 msgr2=0x7f5308005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5321fd3700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c144600 0x7f5308005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5321fd3700 1 -- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308004850 msgr2=0x7f5308004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5321fd3700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308004850 0x7f5308004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5321fd3700 1 -- 172.21.15.89:0/1700145728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53080035b0 con 0x7f5308003a30 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5321fd3700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5308003a30 0x7f5308003e50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5318009890 tx=0x7f5308005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4966c54cceac7f80 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531800c070 con 0x7f5308003a30 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5318015a00 con 0x7f5308003a30 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.723+0000 7f5320fd1700 1 -- 172.21.15.89:0/1700145728 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318015d90 con 0x7f5308003a30 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.724+0000 7f5324237700 1 -- 172.21.15.89:0/1700145728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5308003a30 msgr2=0x7f5308003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.724+0000 7f5324237700 1 --2- 172.21.15.89:0/1700145728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5308003a30 0x7f5308003e50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5318009890 tx=0x7f5308005880 comp rx=0 tx=0).stop 2023-11-24T06:20:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.724+0000 7f53227d4700 1 -- 172.21.15.89:0/1700145728 reap_dead start 2023-11-24T06:20:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.724+0000 7f5324237700 1 -- 172.21.15.89:0/1700145728 shutdown_connections 2023-11-24T06:20:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.724+0000 7f5324237700 1 -- 172.21.15.89:0/1700145728 >> 172.21.15.89:0/1700145728 conn(0x7f531c0d0b40 msgr2=0x7f531c0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.724+0000 7f5324237700 1 -- 172.21.15.89:0/1700145728 shutdown_connections 2023-11-24T06:20:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.724+0000 7f5324237700 1 -- 172.21.15.89:0/1700145728 wait complete. 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.725+0000 7f5324237700 1 Processor -- start 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.725+0000 7f5324237700 1 -- start start 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5324237700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308003a30 0x7f531c0cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5324237700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f531c15a6d0 0x7f531c159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5324237700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c13e910 0x7f531c159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5324237700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c13e480 con 0x7f531c15a6d0 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5324237700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c159fb0 con 0x7f5308003a30 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5324237700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c13c420 con 0x7f531c13e910 2023-11-24T06:20:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f531c15a6d0 0x7f531c159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f531c15a6d0 0x7f531c159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37582/0 (socket says 172.21.15.89:37582) 2023-11-24T06:20:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 -- 172.21.15.89:0/59906437 learned_addr learned my addr 172.21.15.89:0/59906437 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 -- 172.21.15.89:0/59906437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c13e910 msgr2=0x7f531c159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c13e910 0x7f531c159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 -- 172.21.15.89:0/59906437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308003a30 msgr2=0x7f531c0cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5321fd3700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308003a30 0x7f531c0cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308003a30 0x7f531c0cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 -- 172.21.15.89:0/59906437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c13c700 con 0x7f531c15a6d0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5321fd3700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308003a30 0x7f531c0cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53217d2700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f531c15a6d0 0x7f531c159530 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f530c0072c0 tx=0x7f530c0075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53127fc700 1 -- 172.21.15.89:0/59906437 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c0077b0 con 0x7f531c15a6d0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53127fc700 1 -- 172.21.15.89:0/59906437 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c00f850 con 0x7f531c15a6d0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f53127fc700 1 -- 172.21.15.89:0/59906437 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c00fc00 con 0x7f531c15a6d0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c13e110 con 0x7f531c15a6d0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.726+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c13ddc0 con 0x7f531c15a6d0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.727+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531c067f60 con 0x7f531c15a6d0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.728+0000 7f53127fc700 1 -- 172.21.15.89:0/59906437 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f530c00fd60 con 0x7f531c15a6d0 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.728+0000 7f53127fc700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f530006aed0 0x7f530006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.728+0000 7f5321fd3700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f530006aed0 0x7f530006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.728+0000 7f53127fc700 1 -- 172.21.15.89:0/59906437 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5443+0+0 (secure 0 0 0) 0x7f530c089570 con 0x7f531c15a6d0 2023-11-24T06:20:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.729+0000 7f5321fd3700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f530006aed0 0x7f530006d390 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5318000c00 tx=0x7f531800c000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.733+0000 7f53127fc700 1 -- 172.21.15.89:0/59906437 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f530c054680 con 0x7f531c15a6d0 2023-11-24T06:20:12.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:12.933+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f531c06e100 con 0x7f531c15a6d0 2023-11-24T06:20:13.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.573+0000 7f53127fc700 1 -- 172.21.15.89:0/59906437 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) v1 ==== 169+0+0 (secure 0 0 0) 0x7f530c057ca0 con 0x7f531c15a6d0 2023-11-24T06:20:13.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.576+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f530006aed0 msgr2=0x7f530006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.576+0000 7f5324237700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f530006aed0 0x7f530006d390 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5318000c00 tx=0x7f531800c000 comp rx=0 tx=0).stop 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.576+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f531c15a6d0 msgr2=0x7f531c159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.576+0000 7f5324237700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f531c15a6d0 0x7f531c159530 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f530c0072c0 tx=0x7f530c0075d0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.577+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 shutdown_connections 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.577+0000 7f5324237700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f531c13e910 0x7f531c159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.577+0000 7f5324237700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f530006aed0 0x7f530006d390 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.577+0000 7f5324237700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5308003a30 0x7f531c0cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.577+0000 7f5324237700 1 --2- 172.21.15.89:0/59906437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f531c15a6d0 0x7f531c159530 secure :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f530c0072c0 tx=0x7f530c0075d0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.577+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 >> 172.21.15.89:0/59906437 conn(0x7f531c0d0b40 msgr2=0x7f531c15e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.577+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 shutdown_connections 2023-11-24T06:20:13.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.577+0000 7f5324237700 1 -- 172.21.15.89:0/59906437 wait complete. 2023-11-24T06:20:13.580 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2023-11-24T06:20:13.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T06:20:13.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.720+0000 7f44cf6d2700 1 Processor -- start 2023-11-24T06:20:13.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.720+0000 7f44cf6d2700 1 -- start start 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.721+0000 7f44cf6d2700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44c81587f0 0x7f44c806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.721+0000 7f44cf6d2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f44c815b370 0x7f44c815b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.721+0000 7f44cf6d2700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c806b530 con 0x7f44c80b3c60 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.721+0000 7f44cf6d2700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c80ae5e0 con 0x7f44c81587f0 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.721+0000 7f44cf6d2700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c815be30 con 0x7f44c815b370 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.721+0000 7f44cd46e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44c81587f0 0x7f44c806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.721+0000 7f44cd46e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44c81587f0 0x7f44c806aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37588/0 (socket says 172.21.15.89:37588) 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.721+0000 7f44cd46e700 1 -- 172.21.15.89:0/473539006 learned_addr learned my addr 172.21.15.89:0/473539006 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44ccc6d700 1 --2- 172.21.15.89:0/473539006 >> v2:172.21.15.123:3300/0 conn(0x7f44c815b370 0x7f44c815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44cd46e700 1 -- 172.21.15.89:0/473539006 >> v2:172.21.15.123:3300/0 conn(0x7f44c815b370 msgr2=0x7f44c815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44cd46e700 1 --2- 172.21.15.89:0/473539006 >> v2:172.21.15.123:3300/0 conn(0x7f44c815b370 0x7f44c815b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44cd46e700 1 -- 172.21.15.89:0/473539006 >> v1:172.21.15.123:6789/0 conn(0x7f44c80b3c60 legacy=0x7f44c80af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44cd46e700 1 -- 172.21.15.89:0/473539006 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c815c110 con 0x7f44c81587f0 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44cd46e700 1 --2- 172.21.15.89:0/473539006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c81587f0 0x7f44c806aff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f44b800bb90 tx=0x7f44b800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580eeb4316b4ab4a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44bffff700 1 -- 172.21.15.89:0/473539006 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b800dd10 con 0x7f44c81587f0 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44bffff700 1 -- 172.21.15.89:0/473539006 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b800de70 con 0x7f44c81587f0 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44bffff700 1 -- 172.21.15.89:0/473539006 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8004230 con 0x7f44c81587f0 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.722+0000 7f44cf6d2700 1 -- 172.21.15.89:0/473539006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c81587f0 msgr2=0x7f44c806aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.723+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/473539006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c81587f0 0x7f44c806aff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f44b800bb90 tx=0x7f44b800bea0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.723+0000 7f44cf6d2700 1 -- 172.21.15.89:0/473539006 shutdown_connections 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.723+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/473539006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c81587f0 0x7f44c806aff0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.723+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/473539006 >> v2:172.21.15.123:3300/0 conn(0x7f44c815b370 0x7f44c815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.723+0000 7f44cf6d2700 1 -- 172.21.15.89:0/473539006 >> 172.21.15.89:0/473539006 conn(0x7f44c80d0bf0 msgr2=0x7f44c80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:13.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.723+0000 7f44cf6d2700 1 -- 172.21.15.89:0/473539006 shutdown_connections 2023-11-24T06:20:13.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.723+0000 7f44cf6d2700 1 -- 172.21.15.89:0/473539006 wait complete. 2023-11-24T06:20:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.724+0000 7f44cf6d2700 1 Processor -- start 2023-11-24T06:20:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.724+0000 7f44cf6d2700 1 -- start start 2023-11-24T06:20:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.724+0000 7f44cf6d2700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44c80b3c60 0x7f44c80b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.724+0000 7f44cf6d2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44c81587f0 0x7f44c8145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cf6d2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c815b370 0x7f44c8145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cf6d2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c8153280 con 0x7f44c815b370 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cf6d2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c8155870 con 0x7f44c81587f0 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cf6d2700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c8155b50 con 0x7f44c80b3c60 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c815b370 0x7f44c8145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c815b370 0x7f44c8145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37598/0 (socket says 172.21.15.89:37598) 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 -- 172.21.15.89:0/512431463 learned_addr learned my addr 172.21.15.89:0/512431463 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44ccc6d700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44c81587f0 0x7f44c8145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cd46e700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44c80b3c60 0x7f44c80b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 -- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44c80b3c60 msgr2=0x7f44c80b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44c80b3c60 0x7f44c80b18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 -- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44c81587f0 msgr2=0x7f44c8145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44c81587f0 0x7f44c8145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 -- 172.21.15.89:0/512431463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b800b7f0 con 0x7f44c815b370 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44ccc6d700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44c81587f0 0x7f44c8145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cdc6f700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c815b370 0x7f44c8145640 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f44c000cc10 tx=0x7f44c000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44bdffb700 1 -- 172.21.15.89:0/512431463 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c000abb0 con 0x7f44c815b370 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44bdffb700 1 -- 172.21.15.89:0/512431463 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c000ad10 con 0x7f44c815b370 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c8155e30 con 0x7f44c815b370 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44bdffb700 1 -- 172.21.15.89:0/512431463 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0012660 con 0x7f44c815b370 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.725+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c80b1e30 con 0x7f44c815b370 2023-11-24T06:20:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.726+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c80ae1c0 con 0x7f44c815b370 2023-11-24T06:20:13.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.727+0000 7f44bdffb700 1 -- 172.21.15.89:0/512431463 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f44c00127c0 con 0x7f44c815b370 2023-11-24T06:20:13.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.727+0000 7f44bdffb700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f44ac06aed0 0x7f44ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:13.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.727+0000 7f44cd46e700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f44ac06aed0 0x7f44ac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:13.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.728+0000 7f44bdffb700 1 -- 172.21.15.89:0/512431463 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f44c0089b50 con 0x7f44c815b370 2023-11-24T06:20:13.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.728+0000 7f44cd46e700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f44ac06aed0 0x7f44ac06d390 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f44b8000c00 tx=0x7f44b8013000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:13.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.732+0000 7f44bdffb700 1 -- 172.21.15.89:0/512431463 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44c0054e00 con 0x7f44c815b370 2023-11-24T06:20:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.923+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f44c809ece0 con 0x7f44c815b370 2023-11-24T06:20:13.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.927+0000 7f44bdffb700 1 -- 172.21.15.89:0/512431463 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f44c0016020 con 0x7f44c815b370 2023-11-24T06:20:13.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f44ac06aed0 msgr2=0x7f44ac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:13.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f44ac06aed0 0x7f44ac06d390 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f44b8000c00 tx=0x7f44b8013000 comp rx=0 tx=0).stop 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c815b370 msgr2=0x7f44c8145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c815b370 0x7f44c8145640 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f44c000cc10 tx=0x7f44c000a060 comp rx=0 tx=0).stop 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 shutdown_connections 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44c815b370 0x7f44c8145640 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44c80b3c60 0x7f44c80b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f44ac06aed0 0x7f44ac06d390 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 --2- 172.21.15.89:0/512431463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44c81587f0 0x7f44c8145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 >> 172.21.15.89:0/512431463 conn(0x7f44c80d0bf0 msgr2=0x7f44c80b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 shutdown_connections 2023-11-24T06:20:13.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:13.929+0000 7f44cf6d2700 1 -- 172.21.15.89:0/512431463 wait complete. 2023-11-24T06:20:13.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_create' 2023-11-24T06:20:14.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.065+0000 7fd13ee3a700 1 Processor -- start 2023-11-24T06:20:14.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.065+0000 7fd13ee3a700 1 -- start start 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd13ee3a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd13815e080 0x7fd1380b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd13ee3a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd13806c710 0x7fd1380afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd13ee3a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1380ae430 con 0x7fd1380af4d0 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd13ee3a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1380af0b0 con 0x7fd13806c710 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd13ee3a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380b05a0 con 0x7fd13815e080 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd137fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd1380af4d0 0x7fd13806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41106/0 (socket says 172.21.15.89:41106) 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd137fff700 1 -- 172.21.15.89:0/294822945 learned_addr learned my addr 172.21.15.89:0/294822945 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd13cbd6700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd13815e080 0x7fd1380b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd13d3d7700 1 --2- 172.21.15.89:0/294822945 >> v2:172.21.15.123:3300/0 conn(0x7fd13806c710 0x7fd1380afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928155279 0 0) 0x7fd1380ae430 con 0x7fd1380af4d0 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1280035b0 con 0x7fd1380af4d0 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3192285907 0 0) 0x7fd1280035b0 con 0x7fd1380af4d0 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd13815e080 msgr2=0x7fd1380b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd13815e080 0x7fd1380b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 >> v2:172.21.15.123:3300/0 conn(0x7fd13806c710 msgr2=0x7fd1380afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 --2- 172.21.15.89:0/294822945 >> v2:172.21.15.123:3300/0 conn(0x7fd13806c710 0x7fd1380afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1380ae710 con 0x7fd1380af4d0 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fd124001670 con 0x7fd1380af4d0 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 >> v1:172.21.15.89:6789/0 conn(0x7fd1380af4d0 legacy=0x7fd13806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.066+0000 7fd1377fe700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd128003a30 0x7fd128003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd1377fe700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd13815e080 0x7fd128004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd1377fe700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd128004d90 0x7fd128005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380af0b0 con 0x7fd128004d90 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380b05a0 con 0x7fd128003a30 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12800ba10 con 0x7fd13815e080 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd13d3d7700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd128004d90 0x7fd128005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd13cbd6700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd128003a30 0x7fd128003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd137fff700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd13815e080 0x7fd128004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd13d3d7700 1 -- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd13815e080 msgr2=0x7fd128004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd13d3d7700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd13815e080 0x7fd128004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd13d3d7700 1 -- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd128003a30 msgr2=0x7fd128003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd13d3d7700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd128003a30 0x7fd128003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd13d3d7700 1 -- 172.21.15.89:0/294822945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1280035b0 con 0x7fd128004d90 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd13d3d7700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd128004d90 0x7fd128005210 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e170 tx=0x7fd128005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b64adc7b03e2e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13000ec90 con 0x7fd128004d90 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13000f040 con 0x7fd128004d90 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.067+0000 7fd1377fe700 1 -- 172.21.15.89:0/294822945 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130014480 con 0x7fd128004d90 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.068+0000 7fd13ee3a700 1 -- 172.21.15.89:0/294822945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd128004d90 msgr2=0x7fd128005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.068+0000 7fd13ee3a700 1 --2- 172.21.15.89:0/294822945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd128004d90 0x7fd128005210 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e170 tx=0x7fd128005880 comp rx=0 tx=0).stop 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.068+0000 7fd13d3d7700 1 -- 172.21.15.89:0/294822945 reap_dead start 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.068+0000 7fd13ee3a700 1 -- 172.21.15.89:0/294822945 shutdown_connections 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.068+0000 7fd13ee3a700 1 -- 172.21.15.89:0/294822945 >> 172.21.15.89:0/294822945 conn(0x7fd138071b90 msgr2=0x7fd138071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.068+0000 7fd13ee3a700 1 -- 172.21.15.89:0/294822945 shutdown_connections 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.068+0000 7fd13ee3a700 1 -- 172.21.15.89:0/294822945 wait complete. 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.069+0000 7fd13ee3a700 1 Processor -- start 2023-11-24T06:20:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.069+0000 7fd13ee3a700 1 -- start start 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd13ee3a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd128004d90 0x7fd138160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd13ee3a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd138152620 0x7fd138151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd13ee3a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1380cec30 0x7fd138157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd13ee3a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd138161220 con 0x7fd138152620 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd13ee3a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1381523d0 con 0x7fd1380cec30 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd13ee3a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13815f980 con 0x7fd128004d90 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd138152620 0x7fd138151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd138152620 0x7fd138151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37616/0 (socket says 172.21.15.89:37616) 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 -- 172.21.15.89:0/1174170316 learned_addr learned my addr 172.21.15.89:0/1174170316 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd13cbd6700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd128004d90 0x7fd138160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd13d3d7700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1380cec30 0x7fd138157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 -- 172.21.15.89:0/1174170316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd128004d90 msgr2=0x7fd138160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd128004d90 0x7fd138160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 -- 172.21.15.89:0/1174170316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1380cec30 msgr2=0x7fd138157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1380cec30 0x7fd138157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 -- 172.21.15.89:0/1174170316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13815fc40 con 0x7fd138152620 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.070+0000 7fd137fff700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd138152620 0x7fd138151e90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd1240072c0 tx=0x7fd124007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.071+0000 7fd1357fa700 1 -- 172.21.15.89:0/1174170316 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12400f870 con 0x7fd138152620 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.071+0000 7fd1357fa700 1 -- 172.21.15.89:0/1174170316 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12400f9f0 con 0x7fd138152620 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.071+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13815ff20 con 0x7fd138152620 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.071+0000 7fd1357fa700 1 -- 172.21.15.89:0/1174170316 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12400fda0 con 0x7fd138152620 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.071+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1381604e0 con 0x7fd138152620 2023-11-24T06:20:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.071+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1380753f0 con 0x7fd138152620 2023-11-24T06:20:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.072+0000 7fd1357fa700 1 -- 172.21.15.89:0/1174170316 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd12401d460 con 0x7fd138152620 2023-11-24T06:20:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.072+0000 7fd1357fa700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11c06aed0 0x7fd11c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.072+0000 7fd1357fa700 1 -- 172.21.15.89:0/1174170316 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd1240893d0 con 0x7fd138152620 2023-11-24T06:20:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.073+0000 7fd13cbd6700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11c06aed0 0x7fd11c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.073+0000 7fd13cbd6700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11c06aed0 0x7fd11c06d390 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd138152a90 tx=0x7fd12c018040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.077+0000 7fd1357fa700 1 -- 172.21.15.89:0/1174170316 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd124054720 con 0x7fd138152620 2023-11-24T06:20:14.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.343+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_create"]} v 0) v1 -- 0x7fd138067f60 con 0x7fd138152620 2023-11-24T06:20:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.586+0000 7fd1357fa700 1 -- 172.21.15.89:0/1174170316 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_create"]}]=0 v106) v1 ==== 135+0+0 (secure 0 0 0) 0x7fd124057d40 con 0x7fd138152620 2023-11-24T06:20:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11c06aed0 msgr2=0x7fd11c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11c06aed0 0x7fd11c06d390 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd138152a90 tx=0x7fd12c018040 comp rx=0 tx=0).stop 2023-11-24T06:20:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd138152620 msgr2=0x7fd138151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd138152620 0x7fd138151e90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd1240072c0 tx=0x7fd124007680 comp rx=0 tx=0).stop 2023-11-24T06:20:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 shutdown_connections 2023-11-24T06:20:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd11c06aed0 0x7fd11c06d390 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1380cec30 0x7fd138157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd128004d90 0x7fd138160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 --2- 172.21.15.89:0/1174170316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd138152620 0x7fd138151e90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 >> 172.21.15.89:0/1174170316 conn(0x7fd138071b90 msgr2=0x7fd138159250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:14.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 shutdown_connections 2023-11-24T06:20:14.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.589+0000 7fd13ee3a700 1 -- 172.21.15.89:0/1174170316 wait complete. 2023-11-24T06:20:14.602 INFO:tasks.cephfs_test_runner:That the volume can be created and then cleans up ... ok 2023-11-24T06:20:14.603 INFO:tasks.cephfs_test_runner:Starting test: test_volume_info (tasks.cephfs.test_volumes.TestVolumes) Tests the 'fs volume info' command 2023-11-24T06:20:14.603 INFO:tasks.cephfs_test_runner:test_volume_info (tasks.cephfs.test_volumes.TestVolumes) 2023-11-24T06:20:14.603 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.721+0000 7fea75788700 1 Processor -- start 2023-11-24T06:20:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.721+0000 7fea75788700 1 -- start start 2023-11-24T06:20:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea75788700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea700d0870 0x7fea700d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea75788700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea700d1210 con 0x7fea7015e1e0 2023-11-24T06:20:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea75788700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea700d20b0 con 0x7fea700b6ef0 2023-11-24T06:20:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea75788700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea700d3150 con 0x7fea700d0870 2023-11-24T06:20:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6effd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fea7015e1e0 0x7fea700d34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41126/0 (socket says 172.21.15.89:41126) 2023-11-24T06:20:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6effd700 1 -- 172.21.15.89:0/680160358 learned_addr learned my addr 172.21.15.89:0/680160358 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54891739 0 0) 0x7fea700d1210 con 0x7fea7015e1e0 2023-11-24T06:20:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6e7fc700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea700d0870 0x7fea700d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea580035b0 con 0x7fea7015e1e0 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2751321985 0 0) 0x7fea580035b0 con 0x7fea7015e1e0 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea700d0870 msgr2=0x7fea700d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea700d0870 0x7fea700d0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 >> v1:172.21.15.123:6789/0 conn(0x7fea700b6ef0 legacy=0x7fea700b7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700d2390 con 0x7fea7015e1e0 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fea64001670 con 0x7fea7015e1e0 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 >> v1:172.21.15.89:6789/0 conn(0x7fea7015e1e0 legacy=0x7fea700d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.722+0000 7fea6dffb700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea58003a30 0x7fea58003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6dffb700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea700d0870 0x7fea58004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6dffb700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea58004d90 0x7fea58005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700d3150 con 0x7fea58003a30 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5800bb50 con 0x7fea58004d90 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5800bd10 con 0x7fea700d0870 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6effd700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea58003a30 0x7fea58003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6e7fc700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea700d0870 0x7fea58004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6f7fe700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea58004d90 0x7fea58005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6effd700 1 -- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea700d0870 msgr2=0x7fea58004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6effd700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea700d0870 0x7fea58004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6effd700 1 -- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea58004d90 msgr2=0x7fea58005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6effd700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea58004d90 0x7fea58005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6effd700 1 -- 172.21.15.89:0/680160358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580035b0 con 0x7fea58003a30 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6f7fe700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea58004d90 0x7fea58005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6effd700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea58003a30 0x7fea58003e50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fea6400c210 tx=0x7fea58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716325be02fadf79 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64007c10 con 0x7fea58003a30 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6400cdb0 con 0x7fea58003a30 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.723+0000 7fea6dffb700 1 -- 172.21.15.89:0/680160358 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64005ba0 con 0x7fea58003a30 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.724+0000 7fea75788700 1 -- 172.21.15.89:0/680160358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea58003a30 msgr2=0x7fea58003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.724+0000 7fea75788700 1 --2- 172.21.15.89:0/680160358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea58003a30 0x7fea58003e50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fea6400c210 tx=0x7fea58005880 comp rx=0 tx=0).stop 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.724+0000 7fea6f7fe700 1 -- 172.21.15.89:0/680160358 reap_dead start 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.724+0000 7fea75788700 1 -- 172.21.15.89:0/680160358 shutdown_connections 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.724+0000 7fea75788700 1 -- 172.21.15.89:0/680160358 >> 172.21.15.89:0/680160358 conn(0x7fea700b5b50 msgr2=0x7fea700b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.724+0000 7fea75788700 1 -- 172.21.15.89:0/680160358 shutdown_connections 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.724+0000 7fea75788700 1 -- 172.21.15.89:0/680160358 wait complete. 2023-11-24T06:20:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.725+0000 7fea75788700 1 Processor -- start 2023-11-24T06:20:14.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.725+0000 7fea75788700 1 -- start start 2023-11-24T06:20:14.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea75788700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea58003a30 0x7fea700cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea75788700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea700b8a50 0x7fea7014d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea75788700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700b4a50 0x7fea700b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea75788700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700b5410 con 0x7fea700b4a50 2023-11-24T06:20:14.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea75788700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7015ed70 con 0x7fea700b8a50 2023-11-24T06:20:14.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea75788700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7015e740 con 0x7fea58003a30 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700b4a50 0x7fea700b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700b4a50 0x7fea700b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37638/0 (socket says 172.21.15.89:37638) 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 -- 172.21.15.89:0/4069107697 learned_addr learned my addr 172.21.15.89:0/4069107697 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6e7fc700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea700b8a50 0x7fea7014d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6effd700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea58003a30 0x7fea700cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 -- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea58003a30 msgr2=0x7fea700cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea58003a30 0x7fea700cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 -- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea700b8a50 msgr2=0x7fea7014d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea700b8a50 0x7fea7014d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 -- 172.21.15.89:0/4069107697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700b8ed0 con 0x7fea700b4a50 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6e7fc700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea700b8a50 0x7fea7014d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:14.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea6f7fe700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700b4a50 0x7fea700b4ed0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fea70072830 tx=0x7fea6800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea577fe700 1 -- 172.21.15.89:0/4069107697 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68003c10 con 0x7fea700b4a50 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.726+0000 7fea577fe700 1 -- 172.21.15.89:0/4069107697 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea68003d90 con 0x7fea700b4a50 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.727+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea700b91a0 con 0x7fea700b4a50 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.727+0000 7fea577fe700 1 -- 172.21.15.89:0/4069107697 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68005310 con 0x7fea700b4a50 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.727+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7015ea00 con 0x7fea700b4a50 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.727+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7015ebc0 con 0x7fea700b4a50 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.728+0000 7fea577fe700 1 -- 172.21.15.89:0/4069107697 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fea68005470 con 0x7fea700b4a50 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.728+0000 7fea577fe700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fea5c06aed0 0x7fea5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.729+0000 7fea6effd700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fea5c06aed0 0x7fea5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.729+0000 7fea577fe700 1 -- 172.21.15.89:0/4069107697 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fea680892e0 con 0x7fea700b4a50 2023-11-24T06:20:14.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.729+0000 7fea6effd700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fea5c06aed0 0x7fea5c06d390 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fea64001930 tx=0x7fea64007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:14.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.733+0000 7fea577fe700 1 -- 172.21.15.89:0/4069107697 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea680546c0 con 0x7fea700b4a50 2023-11-24T06:20:14.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.923+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea7015eee0 con 0x7fea700b4a50 2023-11-24T06:20:14.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.924+0000 7fea577fe700 1 -- 172.21.15.89:0/4069107697 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+13082 (secure 0 0 0) 0x7fea68001b50 con 0x7fea700b4a50 2023-11-24T06:20:14.926 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:14.926 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":45,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:13.572102+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6835/2053513777":"2023-11-25T06:19:59.514768+0000","172.21.15.123:6835/1601724762":"2023-11-25T06:20:10.448244+0000","172.21.15.123:6833/1601724762":"2023-11-25T06:20:10.448244+0000","172.21.15.89:6837/2053513777":"2023-11-25T06:19:59.514768+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:14.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fea5c06aed0 msgr2=0x7fea5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fea5c06aed0 0x7fea5c06d390 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fea64001930 tx=0x7fea64007000 comp rx=0 tx=0).stop 2023-11-24T06:20:14.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700b4a50 msgr2=0x7fea700b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:14.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700b4a50 0x7fea700b4ed0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fea70072830 tx=0x7fea6800ee70 comp rx=0 tx=0).stop 2023-11-24T06:20:14.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 shutdown_connections 2023-11-24T06:20:14.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea700b4a50 0x7fea700b4ed0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea58003a30 0x7fea700cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fea5c06aed0 0x7fea5c06d390 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 --2- 172.21.15.89:0/4069107697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea700b8a50 0x7fea7014d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:14.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 >> 172.21.15.89:0/4069107697 conn(0x7fea700b5b50 msgr2=0x7fea7015c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:14.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 shutdown_connections 2023-11-24T06:20:14.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:14.926+0000 7fea75788700 1 -- 172.21.15.89:0/4069107697 wait complete. 2023-11-24T06:20:14.940 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:14.940 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.061+0000 7f8485177700 1 Processor -- start 2023-11-24T06:20:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.061+0000 7f8485177700 1 -- start start 2023-11-24T06:20:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f8485177700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f84800746a0 0x7f84800b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f8485177700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84800afc90 0x7f848006d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f8485177700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84800b3c60 con 0x7f848015e160 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f8485177700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84800b4500 con 0x7f84800746a0 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f8485177700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84800b58c0 con 0x7f84800afc90 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847ed9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f848015e160 0x7f8480073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41146/0 (socket says 172.21.15.89:41146) 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847ed9d700 1 -- 172.21.15.89:0/2833263687 learned_addr learned my addr 172.21.15.89:0/2833263687 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847e59c700 1 --2- 172.21.15.89:0/2833263687 >> v2:172.21.15.123:3300/0 conn(0x7f84800746a0 0x7f84800b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847f59e700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84800afc90 0x7f848006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755966636 0 0) 0x7f84800b3c60 con 0x7f848015e160 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84680035b0 con 0x7f848015e160 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3759142011 0 0) 0x7f84680035b0 con 0x7f848015e160 2023-11-24T06:20:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84800afc90 msgr2=0x7f848006d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84800afc90 0x7f848006d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 >> v2:172.21.15.123:3300/0 conn(0x7f84800746a0 msgr2=0x7f84800b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 --2- 172.21.15.89:0/2833263687 >> v2:172.21.15.123:3300/0 conn(0x7f84800746a0 0x7f84800b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84800b0120 con 0x7f848015e160 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f8470001670 con 0x7f848015e160 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847e59c700 1 --2- 172.21.15.89:0/2833263687 >> v2:172.21.15.123:3300/0 conn(0x7f84800746a0 0x7f84800b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 >> v1:172.21.15.89:6789/0 conn(0x7f848015e160 legacy=0x7f8480073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8468003a00 0x7f8468003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84800afc90 0x7f8468004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468004d90 0x7f8468005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84800b4500 con 0x7f8468003a00 2023-11-24T06:20:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84800b58c0 con 0x7f8468004d90 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846800ba10 con 0x7f84800afc90 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.062+0000 7f847ed9d700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8468003a00 0x7f8468003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847e59c700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84800afc90 0x7f8468004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847f59e700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468004d90 0x7f8468005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847ed9d700 1 -- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84800afc90 msgr2=0x7f8468004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847ed9d700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84800afc90 0x7f8468004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847ed9d700 1 -- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468004d90 msgr2=0x7f8468005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847ed9d700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468004d90 0x7f8468005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847ed9d700 1 -- 172.21.15.89:0/2833263687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84680035b0 con 0x7f8468003a00 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847f59e700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468004d90 0x7f8468005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847ed9d700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8468003a00 0x7f8468003e20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f847000c370 tx=0x7f8468005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4156a6c43a63ca46 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470007c60 con 0x7f8468003a00 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8470005960 con 0x7f8468003a00 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847dd9b700 1 -- 172.21.15.89:0/2833263687 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470005d10 con 0x7f8468003a00 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f8485177700 1 -- 172.21.15.89:0/2833263687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8468003a00 msgr2=0x7f8468003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f8485177700 1 --2- 172.21.15.89:0/2833263687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8468003a00 0x7f8468003e20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f847000c370 tx=0x7f8468005880 comp rx=0 tx=0).stop 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.063+0000 7f847f59e700 1 -- 172.21.15.89:0/2833263687 reap_dead start 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.064+0000 7f8485177700 1 -- 172.21.15.89:0/2833263687 shutdown_connections 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.064+0000 7f8485177700 1 -- 172.21.15.89:0/2833263687 >> 172.21.15.89:0/2833263687 conn(0x7f84800d0bf0 msgr2=0x7f84800d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.064+0000 7f8485177700 1 -- 172.21.15.89:0/2833263687 shutdown_connections 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.064+0000 7f8485177700 1 -- 172.21.15.89:0/2833263687 wait complete. 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.065+0000 7f8485177700 1 Processor -- start 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.065+0000 7f8485177700 1 -- start start 2023-11-24T06:20:15.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f8485177700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468003a00 0x7f84800c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f8485177700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f848014a4e0 0x7f84800ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f8485177700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f848014ba20 0x7f848014bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f8485177700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848014c7a0 con 0x7f848014a4e0 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f8485177700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848014df60 con 0x7f8468003a00 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f8485177700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848014e240 con 0x7f848014ba20 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f848014a4e0 0x7f84800ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f848014a4e0 0x7f84800ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37656/0 (socket says 172.21.15.89:37656) 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 -- 172.21.15.89:0/1171432672 learned_addr learned my addr 172.21.15.89:0/1171432672 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847f59e700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f848014ba20 0x7f848014bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847ed9d700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468003a00 0x7f84800c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 -- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f848014ba20 msgr2=0x7f848014bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f848014ba20 0x7f848014bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 -- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468003a00 msgr2=0x7f84800c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468003a00 0x7f84800c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 -- 172.21.15.89:0/1171432672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848014e520 con 0x7f848014a4e0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847f59e700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f848014ba20 0x7f848014bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f847e59c700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f848014a4e0 0x7f84800ca2e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8474011c20 tx=0x7f8474011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.066+0000 7f84677fe700 1 -- 172.21.15.89:0/1171432672 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847400fab0 con 0x7f848014a4e0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.067+0000 7f84677fe700 1 -- 172.21.15.89:0/1171432672 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847400fc10 con 0x7f848014a4e0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.067+0000 7f84677fe700 1 -- 172.21.15.89:0/1171432672 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474004020 con 0x7f848014a4e0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.067+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848014e800 con 0x7f848014a4e0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.067+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848014cb00 con 0x7f848014a4e0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.068+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848014e0a0 con 0x7f848014a4e0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.068+0000 7f84677fe700 1 -- 172.21.15.89:0/1171432672 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8474004180 con 0x7f848014a4e0 2023-11-24T06:20:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.068+0000 7f84677fe700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f846c06aed0 0x7f846c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.068+0000 7f84677fe700 1 -- 172.21.15.89:0/1171432672 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8474089860 con 0x7f848014a4e0 2023-11-24T06:20:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.068+0000 7f847ed9d700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f846c06aed0 0x7f846c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.069+0000 7f847ed9d700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f846c06aed0 0x7f846c06d390 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8470015040 tx=0x7f847000a000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.073+0000 7f84677fe700 1 -- 172.21.15.89:0/1171432672 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f848014e0a0 con 0x7f848014a4e0 2023-11-24T06:20:15.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.263+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f848014e380 con 0x7f848014a4e0 2023-11-24T06:20:15.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.263+0000 7f84677fe700 1 -- 172.21.15.89:0/1171432672 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7f848014e380 con 0x7f848014a4e0 2023-11-24T06:20:15.266 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:15.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f846c06aed0 msgr2=0x7f846c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f846c06aed0 0x7f846c06d390 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8470015040 tx=0x7f847000a000 comp rx=0 tx=0).stop 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f848014a4e0 msgr2=0x7f84800ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f848014a4e0 0x7f84800ca2e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8474011c20 tx=0x7f8474011f30 comp rx=0 tx=0).stop 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 shutdown_connections 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f848014ba20 0x7f848014bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f846c06aed0 0x7f846c06d390 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8468003a00 0x7f84800c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 --2- 172.21.15.89:0/1171432672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f848014a4e0 0x7f84800ca2e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 >> 172.21.15.89:0/1171432672 conn(0x7f84800d0bf0 msgr2=0x7f848006e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 shutdown_connections 2023-11-24T06:20:15.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.266+0000 7f8485177700 1 -- 172.21.15.89:0/1171432672 wait complete. 2023-11-24T06:20:15.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_info' 2023-11-24T06:20:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.398+0000 7fb77ca5a700 1 Processor -- start 2023-11-24T06:20:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.398+0000 7fb77ca5a700 1 -- start start 2023-11-24T06:20:15.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.399+0000 7fb77ca5a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb77815bdd0 0x7fb7781604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.399+0000 7fb77ca5a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780d1c70 0x7fb7780d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.399+0000 7fb77ca5a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780b22d0 con 0x7fb7780b3370 2023-11-24T06:20:15.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.399+0000 7fb77ca5a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7780b2f50 con 0x7fb77815bdd0 2023-11-24T06:20:15.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.399+0000 7fb77ca5a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780d2640 con 0x7fb7780d1c70 2023-11-24T06:20:15.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.399+0000 7fb776d9d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780d1c70 0x7fb7780d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.399+0000 7fb77659c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb77815bdd0 0x7fb7781604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb776d9d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780d1c70 0x7fb7780d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:46542/0 (socket says 172.21.15.89:46542) 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb776d9d700 1 -- 172.21.15.89:0/1249427007 learned_addr learned my addr 172.21.15.89:0/1249427007 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb776d9d700 1 -- 172.21.15.89:0/1249427007 >> v2:172.21.15.123:3300/0 conn(0x7fb77815bdd0 msgr2=0x7fb7781604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb776d9d700 1 --2- 172.21.15.89:0/1249427007 >> v2:172.21.15.123:3300/0 conn(0x7fb77815bdd0 0x7fb7781604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb776d9d700 1 -- 172.21.15.89:0/1249427007 >> v1:172.21.15.123:6789/0 conn(0x7fb7780b3370 legacy=0x7fb7780b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb776d9d700 1 -- 172.21.15.89:0/1249427007 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780b25b0 con 0x7fb7780d1c70 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb776d9d700 1 --2- 172.21.15.89:0/1249427007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780d1c70 0x7fb7780d2100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb770009e50 tx=0x7fb77000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9bc29fb19e2f2f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb77559a700 1 -- 172.21.15.89:0/1249427007 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77000e070 con 0x7fb7780d1c70 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb77559a700 1 -- 172.21.15.89:0/1249427007 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77000b910 con 0x7fb7780d1c70 2023-11-24T06:20:15.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.400+0000 7fb77559a700 1 -- 172.21.15.89:0/1249427007 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77000bcc0 con 0x7fb7780d1c70 2023-11-24T06:20:15.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.401+0000 7fb77ca5a700 1 -- 172.21.15.89:0/1249427007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780d1c70 msgr2=0x7fb7780d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.401+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/1249427007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780d1c70 0x7fb7780d2100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb770009e50 tx=0x7fb77000b330 comp rx=0 tx=0).stop 2023-11-24T06:20:15.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.401+0000 7fb77ca5a700 1 -- 172.21.15.89:0/1249427007 shutdown_connections 2023-11-24T06:20:15.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.401+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/1249427007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780d1c70 0x7fb7780d2100 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.401+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/1249427007 >> v2:172.21.15.123:3300/0 conn(0x7fb77815bdd0 0x7fb7781604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.401+0000 7fb77ca5a700 1 -- 172.21.15.89:0/1249427007 >> 172.21.15.89:0/1249427007 conn(0x7fb7780731a0 msgr2=0x7fb77809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.401+0000 7fb77ca5a700 1 -- 172.21.15.89:0/1249427007 shutdown_connections 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.401+0000 7fb77ca5a700 1 -- 172.21.15.89:0/1249427007 wait complete. 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.402+0000 7fb77ca5a700 1 Processor -- start 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.402+0000 7fb77ca5a700 1 -- start start 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb77ca5a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780b3370 0x7fb778158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb77ca5a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7780d1c70 0x7fb77815e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb77ca5a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb77815bdd0 0x7fb77815ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb77ca5a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b53d0 con 0x7fb7780d1c70 2023-11-24T06:20:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb77ca5a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b55e0 con 0x7fb77815bdd0 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb77ca5a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b58c0 con 0x7fb7780b3370 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb775d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7780d1c70 0x7fb77815e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb775d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7780d1c70 0x7fb77815e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37670/0 (socket says 172.21.15.89:37670) 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb775d9b700 1 -- 172.21.15.89:0/3542873653 learned_addr learned my addr 172.21.15.89:0/3542873653 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb77659c700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780b3370 0x7fb778158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb776d9d700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb77815bdd0 0x7fb77815ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb775d9b700 1 -- 172.21.15.89:0/3542873653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780b3370 msgr2=0x7fb778158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb775d9b700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780b3370 0x7fb778158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb775d9b700 1 -- 172.21.15.89:0/3542873653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb77815bdd0 msgr2=0x7fb77815ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb775d9b700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb77815bdd0 0x7fb77815ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.403+0000 7fb775d9b700 1 -- 172.21.15.89:0/3542873653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770009b30 con 0x7fb7780d1c70 2023-11-24T06:20:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.404+0000 7fb775d9b700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7780d1c70 0x7fb77815e9c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb76800c4b0 tx=0x7fb76800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:15.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.404+0000 7fb766ffd700 1 -- 172.21.15.89:0/3542873653 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7680052d0 con 0x7fb7780d1c70 2023-11-24T06:20:15.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.404+0000 7fb766ffd700 1 -- 172.21.15.89:0/3542873653 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb768005430 con 0x7fb7780d1c70 2023-11-24T06:20:15.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.404+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7780b5ba0 con 0x7fb7780d1c70 2023-11-24T06:20:15.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.404+0000 7fb766ffd700 1 -- 172.21.15.89:0/3542873653 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7680038c0 con 0x7fb7780d1c70 2023-11-24T06:20:15.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.404+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb778158ad0 con 0x7fb7780d1c70 2023-11-24T06:20:15.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.405+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb778067f60 con 0x7fb7780d1c70 2023-11-24T06:20:15.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.405+0000 7fb766ffd700 1 -- 172.21.15.89:0/3542873653 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb768003a20 con 0x7fb7780d1c70 2023-11-24T06:20:15.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.405+0000 7fb766ffd700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75806aed0 0x7fb75806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:15.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.406+0000 7fb766ffd700 1 -- 172.21.15.89:0/3542873653 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb768088860 con 0x7fb7780d1c70 2023-11-24T06:20:15.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.406+0000 7fb77659c700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75806aed0 0x7fb75806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.406+0000 7fb77659c700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75806aed0 0x7fb75806d390 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009ad0 tx=0x7fb76c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.410+0000 7fb766ffd700 1 -- 172.21.15.89:0/3542873653 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb768053b30 con 0x7fb7780d1c70 2023-11-24T06:20:15.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:15.680+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_info"]} v 0) v1 -- 0x7fb7780b7c60 con 0x7fb7780d1c70 2023-11-24T06:20:16.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.591+0000 7fb766ffd700 1 -- 172.21.15.89:0/3542873653 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_info"]}]=0 v108) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb768057150 con 0x7fb7780d1c70 2023-11-24T06:20:16.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75806aed0 msgr2=0x7fb75806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:16.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75806aed0 0x7fb75806d390 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009ad0 tx=0x7fb76c008040 comp rx=0 tx=0).stop 2023-11-24T06:20:16.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7780d1c70 msgr2=0x7fb77815e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:16.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7780d1c70 0x7fb77815e9c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb76800c4b0 tx=0x7fb76800c870 comp rx=0 tx=0).stop 2023-11-24T06:20:16.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 shutdown_connections 2023-11-24T06:20:16.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb75806aed0 0x7fb75806d390 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb77815bdd0 0x7fb77815ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7780b3370 0x7fb778158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 --2- 172.21.15.89:0/3542873653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7780d1c70 0x7fb77815e9c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 >> 172.21.15.89:0/3542873653 conn(0x7fb7780731a0 msgr2=0x7fb7780affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:16.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 shutdown_connections 2023-11-24T06:20:16.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.594+0000 7fb77ca5a700 1 -- 172.21.15.89:0/3542873653 wait complete. 2023-11-24T06:20:16.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T06:20:16.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.738+0000 7fa8c4d23700 1 Processor -- start 2023-11-24T06:20:16.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.738+0000 7fa8c4d23700 1 -- start start 2023-11-24T06:20:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8c4d23700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa8c00746a0 0x7fa8c00b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8c4d23700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8c00afc90 0x7fa8c006d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8c4d23700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c00b3c60 con 0x7fa8c015e160 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8c4d23700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c00b4500 con 0x7fa8c00afc90 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8c4d23700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c00b58c0 con 0x7fa8c00746a0 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8bed9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8c00afc90 0x7fa8c006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8bed9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8c00afc90 0x7fa8c006d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37680/0 (socket says 172.21.15.89:37680) 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8bed9d700 1 -- 172.21.15.89:0/2767235037 learned_addr learned my addr 172.21.15.89:0/2767235037 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8bdd9b700 1 --2- 172.21.15.89:0/2767235037 >> v2:172.21.15.123:3300/0 conn(0x7fa8c00746a0 0x7fa8c00b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8bed9d700 1 -- 172.21.15.89:0/2767235037 >> v2:172.21.15.123:3300/0 conn(0x7fa8c00746a0 msgr2=0x7fa8c00b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8bed9d700 1 --2- 172.21.15.89:0/2767235037 >> v2:172.21.15.123:3300/0 conn(0x7fa8c00746a0 0x7fa8c00b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8bed9d700 1 -- 172.21.15.89:0/2767235037 >> v1:172.21.15.123:6789/0 conn(0x7fa8c015e160 legacy=0x7fa8c0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.739+0000 7fa8bed9d700 1 -- 172.21.15.89:0/2767235037 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c006bdc0 con 0x7fa8c00afc90 2023-11-24T06:20:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8bed9d700 1 --2- 172.21.15.89:0/2767235037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c00afc90 0x7fa8c006d930 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa8c00d0880 tx=0x7fa8b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9e0d5f04881e31 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8bd59a700 1 -- 172.21.15.89:0/2767235037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b800e070 con 0x7fa8c00afc90 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8bd59a700 1 -- 172.21.15.89:0/2767235037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b800b9c0 con 0x7fa8c00afc90 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8bd59a700 1 -- 172.21.15.89:0/2767235037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b800bd70 con 0x7fa8c00afc90 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8c4d23700 1 -- 172.21.15.89:0/2767235037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c00afc90 msgr2=0x7fa8c006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/2767235037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c00afc90 0x7fa8c006d930 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa8c00d0880 tx=0x7fa8b800b330 comp rx=0 tx=0).stop 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8c4d23700 1 -- 172.21.15.89:0/2767235037 shutdown_connections 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/2767235037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c00afc90 0x7fa8c006d930 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/2767235037 >> v2:172.21.15.123:3300/0 conn(0x7fa8c00746a0 0x7fa8c00b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8c4d23700 1 -- 172.21.15.89:0/2767235037 >> 172.21.15.89:0/2767235037 conn(0x7fa8c00d0bf0 msgr2=0x7fa8c00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8c4d23700 1 -- 172.21.15.89:0/2767235037 shutdown_connections 2023-11-24T06:20:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.740+0000 7fa8c4d23700 1 -- 172.21.15.89:0/2767235037 wait complete. 2023-11-24T06:20:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8c4d23700 1 Processor -- start 2023-11-24T06:20:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8c4d23700 1 -- start start 2023-11-24T06:20:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8c4d23700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8c00746a0 0x7fa8c00725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8c4d23700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8c00afc90 0x7fa8c01f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8c4d23700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c015e160 0x7fa8c01f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8c4d23700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c0153290 con 0x7fa8c015e160 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8c4d23700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c0072b20 con 0x7fa8c00746a0 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8c4d23700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c00b0f00 con 0x7fa8c00afc90 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.742+0000 7fa8bed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c015e160 0x7fa8c01f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c015e160 0x7fa8c01f76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37686/0 (socket says 172.21.15.89:37686) 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bed9d700 1 -- 172.21.15.89:0/832557642 learned_addr learned my addr 172.21.15.89:0/832557642 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8be59c700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8c00746a0 0x7fa8c00725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bdd9b700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8c00afc90 0x7fa8c01f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bed9d700 1 -- 172.21.15.89:0/832557642 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8c00afc90 msgr2=0x7fa8c01f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bed9d700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8c00afc90 0x7fa8c01f71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bed9d700 1 -- 172.21.15.89:0/832557642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8c00746a0 msgr2=0x7fa8c00725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bed9d700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8c00746a0 0x7fa8c00725e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bed9d700 1 -- 172.21.15.89:0/832557642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8009b30 con 0x7fa8c015e160 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8bed9d700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c015e160 0x7fa8c01f76e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0072ed0 tx=0x7fa8b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8aeffd700 1 -- 172.21.15.89:0/832557642 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8004880 con 0x7fa8c015e160 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8aeffd700 1 -- 172.21.15.89:0/832557642 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b80099b0 con 0x7fa8c015e160 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c01534e0 con 0x7fa8c015e160 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8aeffd700 1 -- 172.21.15.89:0/832557642 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b801c680 con 0x7fa8c015e160 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.743+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c0153aa0 con 0x7fa8c015e160 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.744+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c0153c30 con 0x7fa8c015e160 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.744+0000 7fa8aeffd700 1 -- 172.21.15.89:0/832557642 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa8b801c7e0 con 0x7fa8c015e160 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.745+0000 7fa8aeffd700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8a006aed0 0x7fa8a006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.745+0000 7fa8aeffd700 1 -- 172.21.15.89:0/832557642 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa8b808aeb0 con 0x7fa8c015e160 2023-11-24T06:20:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.745+0000 7fa8be59c700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8a006aed0 0x7fa8a006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.745+0000 7fa8be59c700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8a006aed0 0x7fa8a006d390 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4007fd0 tx=0x7fa8b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.749+0000 7fa8aeffd700 1 -- 172.21.15.89:0/832557642 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8b8056290 con 0x7fa8c015e160 2023-11-24T06:20:16.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.939+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa8c0068fb0 con 0x7fa8c015e160 2023-11-24T06:20:16.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.942+0000 7fa8aeffd700 1 -- 172.21.15.89:0/832557642 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa8b80598b0 con 0x7fa8c015e160 2023-11-24T06:20:16.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8a006aed0 msgr2=0x7fa8a006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:16.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8a006aed0 0x7fa8a006d390 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4007fd0 tx=0x7fa8b4009040 comp rx=0 tx=0).stop 2023-11-24T06:20:16.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c015e160 msgr2=0x7fa8c01f76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c015e160 0x7fa8c01f76e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0072ed0 tx=0x7fa8b8004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 shutdown_connections 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c015e160 0x7fa8c01f76e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa8a006aed0 0x7fa8a006d390 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8c00746a0 0x7fa8c00725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 --2- 172.21.15.89:0/832557642 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8c00afc90 0x7fa8c01f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 >> 172.21.15.89:0/832557642 conn(0x7fa8c00d0bf0 msgr2=0x7fa8c015c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 shutdown_connections 2023-11-24T06:20:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:16.944+0000 7fa8c4d23700 1 -- 172.21.15.89:0/832557642 wait complete. 2023-11-24T06:20:16.958 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T06:20:16.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:20:17.000 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:20:17.058 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T06:20:17.058 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:17.059 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:17.076 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:20:17.076 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:17.076 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:17.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:20:17.097 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:17.097 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:17.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:20:17.134 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:17.134 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:17.158 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:20:17.158 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T06:20:17.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.291+0000 7f4fb98ea700 1 Processor -- start 2023-11-24T06:20:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.291+0000 7f4fb98ea700 1 -- start start 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb98ea700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb415bef0 0x7f4fb415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb98ea700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4fb41624d0 0x7f4fb4162960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb98ea700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb406dd80 con 0x7f4fb40afe70 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb98ea700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb415f880 con 0x7f4fb41624d0 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb98ea700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb415ed90 con 0x7f4fb415bef0 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb37fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4fb41624d0 0x7f4fb4162960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb37fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4fb41624d0 0x7f4fb4162960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37694/0 (socket says 172.21.15.89:37694) 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb37fe700 1 -- 172.21.15.89:0/3611602358 learned_addr learned my addr 172.21.15.89:0/3611602358 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.292+0000 7f4fb27fc700 1 --1- 172.21.15.89:0/3611602358 >> v1:172.21.15.89:6789/0 conn(0x7f4fb40afe70 0x7f4fb40b4600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41194/0 (socket says 172.21.15.89:41194) 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb2ffd700 1 --2- 172.21.15.89:0/3611602358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb415bef0 0x7f4fb415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb37fe700 1 -- 172.21.15.89:0/3611602358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb415bef0 msgr2=0x7f4fb415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb37fe700 1 --2- 172.21.15.89:0/3611602358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb415bef0 0x7f4fb415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb37fe700 1 -- 172.21.15.89:0/3611602358 >> v1:172.21.15.89:6789/0 conn(0x7f4fb40afe70 legacy=0x7f4fb40b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb1ffb700 1 -- 172.21.15.89:0/3611602358 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205550463 0 0) 0x7f4fb406dd80 con 0x7f4fb40afe70 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb37fe700 1 -- 172.21.15.89:0/3611602358 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb415c2d0 con 0x7f4fb41624d0 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb2ffd700 1 --2- 172.21.15.89:0/3611602358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb415bef0 0x7f4fb415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb37fe700 1 --2- 172.21.15.89:0/3611602358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb41624d0 0x7f4fb4162960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009e50 tx=0x7f4fac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa1da08575cb83d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb1ffb700 1 -- 172.21.15.89:0/3611602358 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac00e070 con 0x7f4fb41624d0 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb1ffb700 1 -- 172.21.15.89:0/3611602358 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac00b9c0 con 0x7f4fb41624d0 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb1ffb700 1 -- 172.21.15.89:0/3611602358 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac00bd70 con 0x7f4fb41624d0 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb98ea700 1 -- 172.21.15.89:0/3611602358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb41624d0 msgr2=0x7f4fb4162960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/3611602358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb41624d0 0x7f4fb4162960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009e50 tx=0x7f4fac00b330 comp rx=0 tx=0).stop 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb98ea700 1 -- 172.21.15.89:0/3611602358 shutdown_connections 2023-11-24T06:20:17.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/3611602358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb41624d0 0x7f4fb4162960 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.293+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/3611602358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb415bef0 0x7f4fb415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.294+0000 7f4fb98ea700 1 -- 172.21.15.89:0/3611602358 >> 172.21.15.89:0/3611602358 conn(0x7f4fb4071ac0 msgr2=0x7f4fb4071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.294+0000 7f4fb98ea700 1 -- 172.21.15.89:0/3611602358 shutdown_connections 2023-11-24T06:20:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.294+0000 7f4fb98ea700 1 -- 172.21.15.89:0/3611602358 wait complete. 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.295+0000 7f4fb98ea700 1 Processor -- start 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.295+0000 7f4fb98ea700 1 -- start start 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb98ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb40afe70 0x7f4fb415d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb98ea700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb415bef0 0x7f4fb40ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb98ea700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb41624d0 0x7f4fb40ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb98ea700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb415c740 con 0x7f4fb40afe70 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb98ea700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40aee10 con 0x7f4fb415bef0 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb98ea700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40af0f0 con 0x7f4fb41624d0 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb40afe70 0x7f4fb415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb40afe70 0x7f4fb415d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37696/0 (socket says 172.21.15.89:37696) 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 -- 172.21.15.89:0/2341005809 learned_addr learned my addr 172.21.15.89:0/2341005809 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb37fe700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb41624d0 0x7f4fb40ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb27fc700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb415bef0 0x7f4fb40ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 -- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb41624d0 msgr2=0x7f4fb40ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb41624d0 0x7f4fb40ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 -- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb415bef0 msgr2=0x7f4fb40ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb415bef0 0x7f4fb40ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 -- 172.21.15.89:0/2341005809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac009b30 con 0x7f4fb40afe70 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb37fe700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb41624d0 0x7f4fb40ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb27fc700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb415bef0 0x7f4fb40ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb2ffd700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb40afe70 0x7f4fb415d890 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800c7f0 tx=0x7f4fa800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4f9bfff700 1 -- 172.21.15.89:0/2341005809 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8011800 con 0x7f4fb40afe70 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4f9bfff700 1 -- 172.21.15.89:0/2341005809 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa8011960 con 0x7f4fb40afe70 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb40af3d0 con 0x7f4fb40afe70 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.296+0000 7f4f9bfff700 1 -- 172.21.15.89:0/2341005809 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8011d10 con 0x7f4fb40afe70 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.297+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb40af990 con 0x7f4fb40afe70 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.297+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb4067f60 con 0x7f4fb40afe70 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.298+0000 7f4f9bfff700 1 -- 172.21.15.89:0/2341005809 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4fa8018360 con 0x7f4fb40afe70 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.298+0000 7f4f9bfff700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4fa006aed0 0x7f4fa006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.298+0000 7f4fb27fc700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4fa006aed0 0x7f4fa006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.298+0000 7f4f9bfff700 1 -- 172.21.15.89:0/2341005809 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4fa8089300 con 0x7f4fb40afe70 2023-11-24T06:20:17.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.299+0000 7f4fb27fc700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4fa006aed0 0x7f4fa006d390 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40afe40 tx=0x7f4fa4007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:17.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.303+0000 7f4f9bfff700 1 -- 172.21.15.89:0/2341005809 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fa8054640 con 0x7f4fb40afe70 2023-11-24T06:20:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.534+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fb40afbb0 con 0x7f4fb40afe70 2023-11-24T06:20:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.534+0000 7f4f9bfff700 1 -- 172.21.15.89:0/2341005809 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f4fa801fe90 con 0x7f4fb40afe70 2023-11-24T06:20:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:17.537 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/503786989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":503786989},{"type":"v1","addr":"172.21.15.123:6834","nonce":503786989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4901,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/271288882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":271288882},{"type":"v1","addr":"172.21.15.123:6835","nonce":271288882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3511736853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3511736853},{"type":"v1","addr":"172.21.15.89:6836","nonce":3511736853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5361,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3128631650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3128631650},{"type":"v1","addr":"172.21.15.89:6837","nonce":3128631650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[]} 2023-11-24T06:20:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4fa006aed0 msgr2=0x7f4fa006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4fa006aed0 0x7f4fa006d390 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40afe40 tx=0x7f4fa4007d30 comp rx=0 tx=0).stop 2023-11-24T06:20:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb40afe70 msgr2=0x7f4fb415d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb40afe70 0x7f4fb415d890 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800c7f0 tx=0x7f4fa800cbb0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 shutdown_connections 2023-11-24T06:20:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb41624d0 0x7f4fb40ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb40afe70 0x7f4fb415d890 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4fa006aed0 0x7f4fa006d390 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 --2- 172.21.15.89:0/2341005809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb415bef0 0x7f4fb40ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 >> 172.21.15.89:0/2341005809 conn(0x7f4fb4071ac0 msgr2=0x7f4fb40b55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 shutdown_connections 2023-11-24T06:20:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.537+0000 7f4fb98ea700 1 -- 172.21.15.89:0/2341005809 wait complete. 2023-11-24T06:20:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2023-11-24T06:20:17.551 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T06:20:17.551 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T06:20:17.551 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T06:20:17.551 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:20:17.552 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T06:20:17.552 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T06:20:17.552 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T06:20:17.552 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:20:17.552 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T06:20:17.552 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T06:20:17.553 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T06:20:17.553 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:20:17.553 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T06:20:17.553 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T06:20:17.553 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T06:20:17.553 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:20:17.561 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T06:20:17.561 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T06:20:17.562 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T06:20:17.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T06:20:17.564 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T06:20:17.564 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T06:20:17.565 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T06:20:17.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T06:20:17.603 INFO:tasks.ceph.mds.a:Started 2023-11-24T06:20:17.603 DEBUG:teuthology.parallel:result is None 2023-11-24T06:20:17.604 INFO:tasks.ceph.mds.c:Started 2023-11-24T06:20:17.604 DEBUG:teuthology.parallel:result is None 2023-11-24T06:20:17.607 INFO:tasks.ceph.mds.b:Started 2023-11-24T06:20:17.607 DEBUG:teuthology.parallel:result is None 2023-11-24T06:20:17.607 INFO:tasks.ceph.mds.d:Started 2023-11-24T06:20:17.608 DEBUG:teuthology.parallel:result is None 2023-11-24T06:20:17.608 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T06:20:17.681 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-11-24T06:20:17.682 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2023-11-24T06:20:17.696 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2023-11-24T06:20:17.696 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2023-11-24T06:20:17.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.734+0000 7f2a10297700 1 Processor -- start 2023-11-24T06:20:17.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.734+0000 7f2a10297700 1 -- start start 2023-11-24T06:20:17.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.735+0000 7f2a10297700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a0806c710 0x7f2a08165b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.735+0000 7f2a10297700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a080af950 con 0x7f2a080adf00 2023-11-24T06:20:17.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.735+0000 7f2a10297700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a080afc30 con 0x7f2a081446a0 2023-11-24T06:20:17.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.735+0000 7f2a10297700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a08144a80 con 0x7f2a0806c710 2023-11-24T06:20:17.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0d832700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2a080adf00 0x7f2a0806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41230/0 (socket says 172.21.15.89:41230) 2023-11-24T06:20:17.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0d832700 1 -- 172.21.15.89:0/4132470843 learned_addr learned my addr 172.21.15.89:0/4132470843 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:17.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0e834700 1 --2- 172.21.15.89:0/4132470843 >> v2:172.21.15.89:3300/0 conn(0x7f2a0806c710 0x7f2a08165b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0e834700 1 -- 172.21.15.89:0/4132470843 >> v1:172.21.15.123:6789/0 conn(0x7f2a081446a0 legacy=0x7f2a080b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0d031700 1 -- 172.21.15.89:0/4132470843 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906944259 0 0) 0x7f2a080afc30 con 0x7f2a081446a0 2023-11-24T06:20:17.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0e834700 1 -- 172.21.15.89:0/4132470843 >> v1:172.21.15.89:6789/0 conn(0x7f2a080adf00 legacy=0x7f2a0806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0e834700 1 -- 172.21.15.89:0/4132470843 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a080aff10 con 0x7f2a0806c710 2023-11-24T06:20:17.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0e834700 1 --2- 172.21.15.89:0/4132470843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a0806c710 0x7f2a08165b70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009d90 tx=0x7f2a0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286e9b33c964e428 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:17.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0d031700 1 -- 172.21.15.89:0/4132470843 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0000e070 con 0x7f2a0806c710 2023-11-24T06:20:17.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0d031700 1 -- 172.21.15.89:0/4132470843 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0000bbd0 con 0x7f2a0806c710 2023-11-24T06:20:17.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.736+0000 7f2a0d031700 1 -- 172.21.15.89:0/4132470843 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00012430 con 0x7f2a0806c710 2023-11-24T06:20:17.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.737+0000 7f2a10297700 1 -- 172.21.15.89:0/4132470843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a0806c710 msgr2=0x7f2a08165b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.737+0000 7f2a10297700 1 --2- 172.21.15.89:0/4132470843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a0806c710 0x7f2a08165b70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009d90 tx=0x7f2a0000b330 comp rx=0 tx=0).stop 2023-11-24T06:20:17.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.737+0000 7f2a10297700 1 -- 172.21.15.89:0/4132470843 shutdown_connections 2023-11-24T06:20:17.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.737+0000 7f2a10297700 1 --2- 172.21.15.89:0/4132470843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a0806c710 0x7f2a08165b70 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.737+0000 7f2a10297700 1 -- 172.21.15.89:0/4132470843 >> 172.21.15.89:0/4132470843 conn(0x7f2a08071b90 msgr2=0x7f2a08071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:17.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.737+0000 7f2a10297700 1 -- 172.21.15.89:0/4132470843 shutdown_connections 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.737+0000 7f2a10297700 1 -- 172.21.15.89:0/4132470843 wait complete. 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.738+0000 7f2a10297700 1 Processor -- start 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.738+0000 7f2a10297700 1 -- start start 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a10297700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a0806c710 0x7f2a0815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a10297700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a080adf00 0x7f2a0813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a10297700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a081446a0 0x7f2a0813fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a10297700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08141fb0 con 0x7f2a081446a0 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a10297700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080cfd60 con 0x7f2a0806c710 2023-11-24T06:20:17.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a10297700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080d0040 con 0x7f2a080adf00 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a081446a0 0x7f2a0813fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a081446a0 0x7f2a0813fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37736/0 (socket says 172.21.15.89:37736) 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 -- 172.21.15.89:0/1116218209 learned_addr learned my addr 172.21.15.89:0/1116218209 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e033700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a0806c710 0x7f2a0815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0d832700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a080adf00 0x7f2a0813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 -- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a080adf00 msgr2=0x7f2a0813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a080adf00 0x7f2a0813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 -- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a0806c710 msgr2=0x7f2a0815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a0806c710 0x7f2a0815a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 -- 172.21.15.89:0/1116218209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a00009a70 con 0x7f2a081446a0 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e033700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a0806c710 0x7f2a0815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:17.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.739+0000 7f2a0e834700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a081446a0 0x7f2a0813fae0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2a00014040 tx=0x7f2a00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:17.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.740+0000 7f29feffd700 1 -- 172.21.15.89:0/1116218209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00012430 con 0x7f2a081446a0 2023-11-24T06:20:17.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.740+0000 7f29feffd700 1 -- 172.21.15.89:0/1116218209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a00020070 con 0x7f2a081446a0 2023-11-24T06:20:17.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.740+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a080d0320 con 0x7f2a081446a0 2023-11-24T06:20:17.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.740+0000 7f29feffd700 1 -- 172.21.15.89:0/1116218209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00007000 con 0x7f2a081446a0 2023-11-24T06:20:17.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.740+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0815aa10 con 0x7f2a081446a0 2023-11-24T06:20:17.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.740+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a080753f0 con 0x7f2a081446a0 2023-11-24T06:20:17.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.741+0000 7f29feffd700 1 -- 172.21.15.89:0/1116218209 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2a00004280 con 0x7f2a081446a0 2023-11-24T06:20:17.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.742+0000 7f29feffd700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f29ec06aed0 0x7f29ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:17.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.742+0000 7f2a0e033700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f29ec06aed0 0x7f29ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:17.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.742+0000 7f29feffd700 1 -- 172.21.15.89:0/1116218209 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2a00022070 con 0x7f2a081446a0 2023-11-24T06:20:17.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.742+0000 7f2a0e033700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f29ec06aed0 0x7f29ec06d390 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000f60 tx=0x7f29f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:17.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.745+0000 7f29feffd700 1 -- 172.21.15.89:0/1116218209 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a00056de0 con 0x7f2a081446a0 2023-11-24T06:20:17.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:17.950+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2a08138ea0 con 0x7f2a081446a0 2023-11-24T06:20:17.989 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:20:17.991 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:20:18.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.612+0000 7f29feffd700 1 -- 172.21.15.89:0/1116218209 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v47) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2a08138ea0 con 0x7f2a081446a0 2023-11-24T06:20:18.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f29ec06aed0 msgr2=0x7f29ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:18.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f29ec06aed0 0x7f29ec06d390 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000f60 tx=0x7f29f8009040 comp rx=0 tx=0).stop 2023-11-24T06:20:18.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a081446a0 msgr2=0x7f2a0813fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:18.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a081446a0 0x7f2a0813fae0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2a00014040 tx=0x7f2a00004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 shutdown_connections 2023-11-24T06:20:18.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a081446a0 0x7f2a0813fae0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f29ec06aed0 0x7f29ec06d390 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a0806c710 0x7f2a0815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 --2- 172.21.15.89:0/1116218209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a080adf00 0x7f2a0813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 >> 172.21.15.89:0/1116218209 conn(0x7f2a08071b90 msgr2=0x7f2a081407c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:18.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 shutdown_connections 2023-11-24T06:20:18.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.615+0000 7f2a10297700 1 -- 172.21.15.89:0/1116218209 wait complete. 2023-11-24T06:20:18.618 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2023-11-24T06:20:18.632 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T06:20:18.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.753+0000 7f78a5add700 1 Processor -- start 2023-11-24T06:20:18.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.753+0000 7f78a5add700 1 -- start start 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f78a5add700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78a00d2240 0x7f78a00b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f78a5add700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00b1d40 con 0x7f78a015ddd0 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f78a5add700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00b28c0 con 0x7f78a015e1b0 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f78a5add700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00b1f90 con 0x7f78a00d2240 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789f7fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f78a015ddd0 0x7f78a00d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41246/0 (socket says 172.21.15.89:41246) 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789f7fe700 1 -- 172.21.15.89:0/358057127 learned_addr learned my addr 172.21.15.89:0/358057127 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989032705 0 0) 0x7f78a00b1d40 con 0x7f78a015ddd0 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78880035b0 con 0x7f78a015ddd0 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789effd700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78a00d2240 0x7f78a00b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3092855485 0 0) 0x7f78880035b0 con 0x7f78a015ddd0 2023-11-24T06:20:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78a00d2240 msgr2=0x7f78a00b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78a00d2240 0x7f78a00b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 >> v1:172.21.15.123:6789/0 conn(0x7f78a015e1b0 legacy=0x7f78a006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00b2270 con 0x7f78a015ddd0 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f7894001670 con 0x7f78a015ddd0 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.754+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 >> v1:172.21.15.89:6789/0 conn(0x7f78a015ddd0 legacy=0x7f78a00d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7888003a30 0x7f7888003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78a00d2240 0x7f7888004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7888004d90 0x7f7888005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00b1f90 con 0x7f7888003a30 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788800bbd0 con 0x7f7888004d90 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788800bd90 con 0x7f78a00d2240 2023-11-24T06:20:18.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789f7fe700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7888003a30 0x7f7888003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789effd700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78a00d2240 0x7f7888004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789ffff700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7888004d90 0x7f7888005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789f7fe700 1 -- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78a00d2240 msgr2=0x7f7888004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789f7fe700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78a00d2240 0x7f7888004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789f7fe700 1 -- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7888004d90 msgr2=0x7f7888005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789f7fe700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7888004d90 0x7f7888005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789f7fe700 1 -- 172.21.15.89:0/358057127 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78880035b0 con 0x7f7888003a30 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789f7fe700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7888003a30 0x7f7888003e50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f789400c240 tx=0x7f7888005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0c90209d62ed04 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894007c10 con 0x7f7888003a30 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789400cde0 con 0x7f7888003a30 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.755+0000 7f789e7fc700 1 -- 172.21.15.89:0/358057127 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894005ba0 con 0x7f7888003a30 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.756+0000 7f78a5add700 1 -- 172.21.15.89:0/358057127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7888003a30 msgr2=0x7f7888003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.756+0000 7f78a5add700 1 --2- 172.21.15.89:0/358057127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7888003a30 0x7f7888003e50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f789400c240 tx=0x7f7888005880 comp rx=0 tx=0).stop 2023-11-24T06:20:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.756+0000 7f789ffff700 1 -- 172.21.15.89:0/358057127 reap_dead start 2023-11-24T06:20:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.756+0000 7f78a5add700 1 -- 172.21.15.89:0/358057127 shutdown_connections 2023-11-24T06:20:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.756+0000 7f78a5add700 1 -- 172.21.15.89:0/358057127 >> 172.21.15.89:0/358057127 conn(0x7f78a00731a0 msgr2=0x7f78a009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.756+0000 7f78a5add700 1 -- 172.21.15.89:0/358057127 shutdown_connections 2023-11-24T06:20:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.756+0000 7f78a5add700 1 -- 172.21.15.89:0/358057127 wait complete. 2023-11-24T06:20:18.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.757+0000 7f78a5add700 1 Processor -- start 2023-11-24T06:20:18.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.758+0000 7f78a5add700 1 -- start start 2023-11-24T06:20:18.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.758+0000 7f78a5add700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7888003a30 0x7f78a014f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:18.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.758+0000 7f78a5add700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78a006c040 0x7f78a00b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:18.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.758+0000 7f78a5add700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00b5720 0x7f78a00b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:18.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.758+0000 7f78a5add700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00b6f90 con 0x7f78a00b5720 2023-11-24T06:20:18.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.758+0000 7f78a5add700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a01f47e0 con 0x7f78a006c040 2023-11-24T06:20:18.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.758+0000 7f78a5add700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a01f4db0 con 0x7f7888003a30 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00b5720 0x7f78a00b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00b5720 0x7f78a00b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37758/0 (socket says 172.21.15.89:37758) 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 -- 172.21.15.89:0/2791411532 learned_addr learned my addr 172.21.15.89:0/2791411532 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789effd700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78a006c040 0x7f78a00b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789f7fe700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7888003a30 0x7f78a014f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 -- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7888003a30 msgr2=0x7f78a014f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7888003a30 0x7f78a014f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 -- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78a006c040 msgr2=0x7f78a00b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78a006c040 0x7f78a00b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 -- 172.21.15.89:0/2791411532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00b2ed0 con 0x7f78a00b5720 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789f7fe700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7888003a30 0x7f78a014f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:18.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f789ffff700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00b5720 0x7f78a00b5ba0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f78a015e590 tx=0x7f789800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f7887fff700 1 -- 172.21.15.89:0/2791411532 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898002070 con 0x7f78a00b5720 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f7887fff700 1 -- 172.21.15.89:0/2791411532 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898003c10 con 0x7f78a00b5720 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a01f5090 con 0x7f78a00b5720 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f7887fff700 1 -- 172.21.15.89:0/2791411532 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789800d990 con 0x7f78a00b5720 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.759+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a01f4ac0 con 0x7f78a00b5720 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.760+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a0067f60 con 0x7f78a00b5720 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.761+0000 7f7887fff700 1 -- 172.21.15.89:0/2791411532 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f789800daf0 con 0x7f78a00b5720 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.761+0000 7f7887fff700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f788c06aed0 0x7f788c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.761+0000 7f7887fff700 1 -- 172.21.15.89:0/2791411532 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f78980893a0 con 0x7f78a00b5720 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.761+0000 7f789f7fe700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f788c06aed0 0x7f788c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.761+0000 7f789f7fe700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f788c06aed0 0x7f788c06d390 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7894001930 tx=0x7f7894007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:18.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:18.765+0000 7f7887fff700 1 -- 172.21.15.89:0/2791411532 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7898054830 con 0x7f78a00b5720 2023-11-24T06:20:19.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.032+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f78a014f600 con 0x7f78a00b5720 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.033+0000 7f7887fff700 1 -- 172.21.15.89:0/2791411532 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7898057e50 con 0x7f78a00b5720 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlZPhnExAAdSzrCYJj/BqJINrd+mxnog==", 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.035 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlAvtyHxAAiLxnKUSDAvkxFP6LHLBkSQ==", 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:19.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlrHqfGhAA7hKW48g57uUyHOiS4gQkOQ==", 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBl4PG2JhAAg8KlX9jNiQ/Ajvrm71BM8g==", 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQASQGBlLUQyBxAApB8r9XPrC9s6zsd2nPQFoQ==", 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.039 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVQGBlFgtPBhAAvSzzgIHzad8zv8vS3fcFQg==", 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAYQGBlNW1sBRAAtOtIYUOAa63AiLIS9EYCFA==", 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.040 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAbQGBlZOIsBRAATWs4ayExofw/56n0ixjgDA==", 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAhQGBlGkIpCRAAWpMWRhHqDSEuyM+DAQLP8Q==", 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.042 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAkQGBli6LfCBAAH5LBKEpbvU48rAyWnyH0Zg==", 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAnQGBl6qpjCBAASFc7C3OIr9G6lSXCNyLk8A==", 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.043 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAqQGBlxmJdCBAANGRCESck0mrD1gVuToRRaw==", 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlwIfyKhAAMZcwOQGAm+XcH+mUKcTiUg==", 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2023-11-24T06:20:19.045 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlxNQDNhAApU+JeKCeMTDo61+wi3ynbw==", 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBluCSzMhAA1hxy9u3cvwTZLG/Xg3kQWQ==", 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:19.046 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlin9NORAAkdx6VeU4rwbgNfVPwwUXFg==", 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:20:19.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQANQGBl+ReMMhAARm9F19thbXVu988u1kapsw==", 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlj+EGEBAAFs6HBnMAVB9ZLdn7WrWpzQ==", 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlsrk+DBAAdgV15cObrTUfehK6bUY8DA==", 2023-11-24T06:20:19.049 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:19.050 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:20:19.050 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2023-11-24T06:20:19.050 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:19.050 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.050 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:19.050 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-11-24T06:20:19.050 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f788c06aed0 msgr2=0x7f788c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f788c06aed0 0x7f788c06d390 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7894001930 tx=0x7f7894007000 comp rx=0 tx=0).stop 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00b5720 msgr2=0x7f78a00b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00b5720 0x7f78a00b5ba0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f78a015e590 tx=0x7f789800d430 comp rx=0 tx=0).stop 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 shutdown_connections 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00b5720 0x7f78a00b5ba0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7888003a30 0x7f78a014f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f788c06aed0 0x7f788c06d390 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 --2- 172.21.15.89:0/2791411532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78a006c040 0x7f78a00b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 >> 172.21.15.89:0/2791411532 conn(0x7f78a00731a0 msgr2=0x7f78a00701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 shutdown_connections 2023-11-24T06:20:19.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.035+0000 7f78a5add700 1 -- 172.21.15.89:0/2791411532 wait complete. 2023-11-24T06:20:19.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.170+0000 7f1ee88e5700 1 Processor -- start 2023-11-24T06:20:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.171+0000 7f1ee88e5700 1 -- start start 2023-11-24T06:20:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.171+0000 7f1ee88e5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee40d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee88e5700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1ee40d0f80 0x7f1ee40b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee88e5700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee40d20b0 con 0x7f1ee40d3a00 2023-11-24T06:20:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee88e5700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee40d3150 con 0x7f1ee40d0f80 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee88e5700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee415e5c0 con 0x7f1ee415e1e0 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee259c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee40d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee2d9d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1ee40d0f80 0x7f1ee40b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee1d9b700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f1ee40d3a00 0x7f1ee40d0870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:45088/0 (socket says 172.21.15.89:45088) 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee1d9b700 1 -- 172.21.15.89:0/2505285574 learned_addr learned my addr 172.21.15.89:0/2505285574 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee259c700 1 -- 172.21.15.89:0/2505285574 >> v2:172.21.15.123:3300/0 conn(0x7f1ee40d0f80 msgr2=0x7f1ee40b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee259c700 1 --2- 172.21.15.89:0/2505285574 >> v2:172.21.15.123:3300/0 conn(0x7f1ee40d0f80 0x7f1ee40b6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee259c700 1 -- 172.21.15.89:0/2505285574 >> v1:172.21.15.123:6789/0 conn(0x7f1ee40d3a00 legacy=0x7f1ee40d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee259c700 1 -- 172.21.15.89:0/2505285574 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee40d2390 con 0x7f1ee415e1e0 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.172+0000 7f1ee2d9d700 1 --2- 172.21.15.89:0/2505285574 >> v2:172.21.15.123:3300/0 conn(0x7f1ee40d0f80 0x7f1ee40b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee259c700 1 --2- 172.21.15.89:0/2505285574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee40d34c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009840 tx=0x7f1ed8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20d0fa0d6c5d93b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee159a700 1 -- 172.21.15.89:0/2505285574 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8011830 con 0x7f1ee415e1e0 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee159a700 1 -- 172.21.15.89:0/2505285574 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed8011990 con 0x7f1ee415e1e0 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee159a700 1 -- 172.21.15.89:0/2505285574 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8011d20 con 0x7f1ee415e1e0 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee88e5700 1 -- 172.21.15.89:0/2505285574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 msgr2=0x7f1ee40d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/2505285574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee40d34c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009840 tx=0x7f1ed8009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee88e5700 1 -- 172.21.15.89:0/2505285574 shutdown_connections 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/2505285574 >> v2:172.21.15.123:3300/0 conn(0x7f1ee40d0f80 0x7f1ee40b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/2505285574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee40d34c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee88e5700 1 -- 172.21.15.89:0/2505285574 >> 172.21.15.89:0/2505285574 conn(0x7f1ee40b5b50 msgr2=0x7f1ee40b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee88e5700 1 -- 172.21.15.89:0/2505285574 shutdown_connections 2023-11-24T06:20:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.173+0000 7f1ee88e5700 1 -- 172.21.15.89:0/2505285574 wait complete. 2023-11-24T06:20:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.175+0000 7f1ee88e5700 1 Processor -- start 2023-11-24T06:20:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.175+0000 7f1ee88e5700 1 -- start start 2023-11-24T06:20:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.175+0000 7f1ee88e5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ee40d0f80 0x7f1ee4159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.175+0000 7f1ee88e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ee40d3a00 0x7f1ee40c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.175+0000 7f1ee88e5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.175+0000 7f1ee88e5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4158d40 con 0x7f1ee40d3a00 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.175+0000 7f1ee88e5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee41595b0 con 0x7f1ee40d0f80 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.175+0000 7f1ee88e5700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4158a60 con 0x7f1ee415e1e0 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ee40d3a00 0x7f1ee40c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ee40d3a00 0x7f1ee40c7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37770/0 (socket says 172.21.15.89:37770) 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 -- 172.21.15.89:0/986060023 learned_addr learned my addr 172.21.15.89:0/986060023 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee259c700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ee40d0f80 0x7f1ee4159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee2d9d700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 -- 172.21.15.89:0/986060023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 msgr2=0x7f1ee415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee415ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 -- 172.21.15.89:0/986060023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ee40d0f80 msgr2=0x7f1ee4159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ee40d0f80 0x7f1ee4159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 -- 172.21.15.89:0/986060023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc009a70 con 0x7f1ee40d3a00 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee1d9b700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ee40d3a00 0x7f1ee40c7190 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400c520 tx=0x7f1ed400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ed2ffd700 1 -- 172.21.15.89:0/986060023 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4005d70 con 0x7f1ee40d3a00 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ed2ffd700 1 -- 172.21.15.89:0/986060023 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed40070d0 con 0x7f1ee40d3a00 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed80094a0 con 0x7f1ee40d3a00 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ed2ffd700 1 -- 172.21.15.89:0/986060023 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4007440 con 0x7f1ee40d3a00 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.176+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee40cbf60 con 0x7f1ee40d3a00 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.177+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee40751e0 con 0x7f1ee40d3a00 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.178+0000 7f1ed2ffd700 1 -- 172.21.15.89:0/986060023 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1ed40075a0 con 0x7f1ee40d3a00 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.178+0000 7f1ed2ffd700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1ec406aed0 0x7f1ec406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.178+0000 7f1ee259c700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1ec406aed0 0x7f1ec406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.178+0000 7f1ed2ffd700 1 -- 172.21.15.89:0/986060023 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1ed4088500 con 0x7f1ee40d3a00 2023-11-24T06:20:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.179+0000 7f1ee259c700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1ec406aed0 0x7f1ec406d390 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8014040 tx=0x7f1ed8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.183+0000 7f1ed2ffd700 1 -- 172.21.15.89:0/986060023 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ed4053a40 con 0x7f1ee40d3a00 2023-11-24T06:20:19.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.372+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ee40b52c0 con 0x7f1ee40d3a00 2023-11-24T06:20:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.372+0000 7f1ed2ffd700 1 -- 172.21.15.89:0/986060023 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+12821 (secure 0 0 0) 0x7f1ed4057060 con 0x7f1ee40d3a00 2023-11-24T06:20:19.377 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:19.377 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":49,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:18.620982+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:19.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1ec406aed0 msgr2=0x7f1ec406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1ec406aed0 0x7f1ec406d390 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8014040 tx=0x7f1ed8008030 comp rx=0 tx=0).stop 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ee40d3a00 msgr2=0x7f1ee40c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ee40d3a00 0x7f1ee40c7190 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400c520 tx=0x7f1ed400c830 comp rx=0 tx=0).stop 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 shutdown_connections 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ee415e1e0 0x7f1ee415ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1ec406aed0 0x7f1ec406d390 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ee40d0f80 0x7f1ee4159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 --2- 172.21.15.89:0/986060023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ee40d3a00 0x7f1ee40c7190 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.374+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 >> 172.21.15.89:0/986060023 conn(0x7f1ee40b5b50 msgr2=0x7f1ee406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.375+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 shutdown_connections 2023-11-24T06:20:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.375+0000 7f1ee88e5700 1 -- 172.21.15.89:0/986060023 wait complete. 2023-11-24T06:20:19.389 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:19.389 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:19.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.510+0000 7f0eb2576700 1 Processor -- start 2023-11-24T06:20:19.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.510+0000 7f0eb2576700 1 -- start start 2023-11-24T06:20:19.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eb2576700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 0x7f0eac0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eb2576700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0eac0afc90 0x7f0eac073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eb2576700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac0b3c60 con 0x7f0eac0744d0 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eb2576700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eac0b4500 con 0x7f0eac0afc90 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eb2576700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac074960 con 0x7f0eac15e160 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eab7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0eac0afc90 0x7f0eac073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eab7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0eac0afc90 0x7f0eac073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37776/0 (socket says 172.21.15.89:37776) 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eab7fe700 1 -- 172.21.15.89:0/1333538928 learned_addr learned my addr 172.21.15.89:0/1333538928 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eabfff700 1 --2- 172.21.15.89:0/1333538928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 0x7f0eac0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eab7fe700 1 -- 172.21.15.89:0/1333538928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 msgr2=0x7f0eac0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:19.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eab7fe700 1 --2- 172.21.15.89:0/1333538928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 0x7f0eac0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eab7fe700 1 -- 172.21.15.89:0/1333538928 >> v1:172.21.15.123:6789/0 conn(0x7f0eac0744d0 legacy=0x7f0eac06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.511+0000 7f0eab7fe700 1 -- 172.21.15.89:0/1333538928 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0b58c0 con 0x7f0eac0afc90 2023-11-24T06:20:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eab7fe700 1 --2- 172.21.15.89:0/1333538928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac073f90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0098f0 tx=0x7f0e9c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3ac15d65f189be server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eaaffd700 1 -- 172.21.15.89:0/1333538928 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c00b8a0 con 0x7f0eac0afc90 2023-11-24T06:20:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eaaffd700 1 -- 172.21.15.89:0/1333538928 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c00ba00 con 0x7f0eac0afc90 2023-11-24T06:20:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eaaffd700 1 -- 172.21.15.89:0/1333538928 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c00bd90 con 0x7f0eac0afc90 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eb2576700 1 -- 172.21.15.89:0/1333538928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 msgr2=0x7f0eac073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eb2576700 1 --2- 172.21.15.89:0/1333538928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac073f90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0098f0 tx=0x7f0e9c009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eb2576700 1 -- 172.21.15.89:0/1333538928 shutdown_connections 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eb2576700 1 --2- 172.21.15.89:0/1333538928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 0x7f0eac0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eb2576700 1 --2- 172.21.15.89:0/1333538928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac073f90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eb2576700 1 -- 172.21.15.89:0/1333538928 >> 172.21.15.89:0/1333538928 conn(0x7f0eac0d0bf0 msgr2=0x7f0eac0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eb2576700 1 -- 172.21.15.89:0/1333538928 shutdown_connections 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.512+0000 7f0eb2576700 1 -- 172.21.15.89:0/1333538928 wait complete. 2023-11-24T06:20:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.514+0000 7f0eb2576700 1 Processor -- start 2023-11-24T06:20:19.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.514+0000 7f0eb2576700 1 -- start start 2023-11-24T06:20:19.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.514+0000 7f0eb2576700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0eac0744d0 0x7f0eac15f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.514+0000 7f0eb2576700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.514+0000 7f0eb2576700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 0x7f0eac15a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.514+0000 7f0eb2576700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac160270 con 0x7f0eac0afc90 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.514+0000 7f0eb2576700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac145f00 con 0x7f0eac0744d0 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.514+0000 7f0eb2576700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac155040 con 0x7f0eac15e160 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac159ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37784/0 (socket says 172.21.15.89:37784) 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 -- 172.21.15.89:0/132761665 learned_addr learned my addr 172.21.15.89:0/132761665 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eabfff700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0eac0744d0 0x7f0eac15f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eb0b13700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 0x7f0eac15a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 -- 172.21.15.89:0/132761665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 msgr2=0x7f0eac15a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 0x7f0eac15a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 -- 172.21.15.89:0/132761665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0eac0744d0 msgr2=0x7f0eac15f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0eac0744d0 0x7f0eac15f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 -- 172.21.15.89:0/132761665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c009550 con 0x7f0eac0afc90 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eab7fe700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac159ad0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009c40 tx=0x7f0e9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0ea8ff9700 1 -- 172.21.15.89:0/132761665 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c004300 con 0x7f0eac0afc90 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0ea8ff9700 1 -- 172.21.15.89:0/132761665 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c01f070 con 0x7f0eac0afc90 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac15f730 con 0x7f0eac0afc90 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0ea8ff9700 1 -- 172.21.15.89:0/132761665 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c0036a0 con 0x7f0eac0afc90 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.515+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac15fcb0 con 0x7f0eac0afc90 2023-11-24T06:20:19.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.516+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac068fb0 con 0x7f0eac0afc90 2023-11-24T06:20:19.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.517+0000 7f0ea8ff9700 1 -- 172.21.15.89:0/132761665 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0e9c021030 con 0x7f0eac0afc90 2023-11-24T06:20:19.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.517+0000 7f0ea8ff9700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0e9806aed0 0x7f0e9806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.517+0000 7f0ea8ff9700 1 -- 172.21.15.89:0/132761665 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e9c08a4c0 con 0x7f0eac0afc90 2023-11-24T06:20:19.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.517+0000 7f0eabfff700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0e9806aed0 0x7f0e9806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.517+0000 7f0eabfff700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0e9806aed0 0x7f0e9806d390 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0006b20 tx=0x7f0ea0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.521+0000 7f0ea8ff9700 1 -- 172.21.15.89:0/132761665 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e9c055950 con 0x7f0eac0afc90 2023-11-24T06:20:19.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.716+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0eac06c140 con 0x7f0eac0afc90 2023-11-24T06:20:19.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.716+0000 7f0ea8ff9700 1 -- 172.21.15.89:0/132761665 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e9c015020 con 0x7f0eac0afc90 2023-11-24T06:20:19.718 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0e9806aed0 msgr2=0x7f0e9806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0e9806aed0 0x7f0e9806d390 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0006b20 tx=0x7f0ea0008040 comp rx=0 tx=0).stop 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 msgr2=0x7f0eac159ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac159ad0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009c40 tx=0x7f0e9c004750 comp rx=0 tx=0).stop 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 shutdown_connections 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0eac15e160 0x7f0eac15a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0e9806aed0 0x7f0e9806d390 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0eac0744d0 0x7f0eac15f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 --2- 172.21.15.89:0/132761665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac0afc90 0x7f0eac159ad0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.718+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 >> 172.21.15.89:0/132761665 conn(0x7f0eac0d0bf0 msgr2=0x7f0eac0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:19.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.719+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 shutdown_connections 2023-11-24T06:20:19.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.719+0000 7f0eb2576700 1 -- 172.21.15.89:0/132761665 wait complete. 2023-11-24T06:20:19.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T06:20:19.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.852+0000 7fe9aee36700 1 Processor -- start 2023-11-24T06:20:19.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.852+0000 7fe9aee36700 1 -- start start 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.852+0000 7fe9aee36700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe9a815e080 0x7fe9a80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.852+0000 7fe9aee36700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9a80af4d0 0x7fe9a80afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.852+0000 7fe9aee36700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a80ae430 con 0x7fe9a806c000 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.852+0000 7fe9aee36700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a80af0b0 con 0x7fe9a80af4d0 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.852+0000 7fe9aee36700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a80b05a0 con 0x7fe9a815e080 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a7fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9a80af4d0 0x7fe9a80afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a7fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9a80af4d0 0x7fe9a80afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37792/0 (socket says 172.21.15.89:37792) 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a7fff700 1 -- 172.21.15.89:0/1186412888 learned_addr learned my addr 172.21.15.89:0/1186412888 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:19.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a7fff700 1 -- 172.21.15.89:0/1186412888 >> v2:172.21.15.123:3300/0 conn(0x7fe9a815e080 msgr2=0x7fe9a80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a7fff700 1 --2- 172.21.15.89:0/1186412888 >> v2:172.21.15.123:3300/0 conn(0x7fe9a815e080 0x7fe9a80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a7fff700 1 -- 172.21.15.89:0/1186412888 >> v1:172.21.15.123:6789/0 conn(0x7fe9a806c000 legacy=0x7fe9a8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a7fff700 1 -- 172.21.15.89:0/1186412888 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a80ae710 con 0x7fe9a80af4d0 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a7fff700 1 --2- 172.21.15.89:0/1186412888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a80af4d0 0x7fe9a80afb90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8072a90 tx=0x7fe994009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d69cc4a4420a3dd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a77fe700 1 -- 172.21.15.89:0/1186412888 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994004230 con 0x7fe9a80af4d0 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a77fe700 1 -- 172.21.15.89:0/1186412888 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe994004390 con 0x7fe9a80af4d0 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.853+0000 7fe9a77fe700 1 -- 172.21.15.89:0/1186412888 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940036a0 con 0x7fe9a80af4d0 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.854+0000 7fe9aee36700 1 -- 172.21.15.89:0/1186412888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a80af4d0 msgr2=0x7fe9a80afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.854+0000 7fe9aee36700 1 --2- 172.21.15.89:0/1186412888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a80af4d0 0x7fe9a80afb90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8072a90 tx=0x7fe994009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:19.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.854+0000 7fe9aee36700 1 -- 172.21.15.89:0/1186412888 shutdown_connections 2023-11-24T06:20:19.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.854+0000 7fe9aee36700 1 --2- 172.21.15.89:0/1186412888 >> v2:172.21.15.123:3300/0 conn(0x7fe9a815e080 0x7fe9a80b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.854+0000 7fe9aee36700 1 --2- 172.21.15.89:0/1186412888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a80af4d0 0x7fe9a80afb90 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.854+0000 7fe9aee36700 1 -- 172.21.15.89:0/1186412888 >> 172.21.15.89:0/1186412888 conn(0x7fe9a8071b90 msgr2=0x7fe9a8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:19.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.854+0000 7fe9aee36700 1 -- 172.21.15.89:0/1186412888 shutdown_connections 2023-11-24T06:20:19.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.854+0000 7fe9aee36700 1 -- 172.21.15.89:0/1186412888 wait complete. 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.855+0000 7fe9aee36700 1 Processor -- start 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.855+0000 7fe9aee36700 1 -- start start 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9aee36700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a806c000 0x7fe9a814a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9aee36700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9a80af4d0 0x7fe9a80b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9aee36700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a815e080 0x7fe9a80b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9aee36700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8153980 con 0x7fe9a806c000 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9aee36700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8158050 con 0x7fe9a815e080 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9aee36700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8158330 con 0x7fe9a80af4d0 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a806c000 0x7fe9a814a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a806c000 0x7fe9a814a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37796/0 (socket says 172.21.15.89:37796) 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 -- 172.21.15.89:0/4208956155 learned_addr learned my addr 172.21.15.89:0/4208956155 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9ad3d3700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a815e080 0x7fe9a80b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9a7fff700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9a80af4d0 0x7fe9a80b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 -- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9a80af4d0 msgr2=0x7fe9a80b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9a80af4d0 0x7fe9a80b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 -- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a815e080 msgr2=0x7fe9a80b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a815e080 0x7fe9a80b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 -- 172.21.15.89:0/4208956155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe994009550 con 0x7fe9a806c000 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9a7fff700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9a80af4d0 0x7fe9a80b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9ad3d3700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a815e080 0x7fe9a80b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9acbd2700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a806c000 0x7fe9a814a360 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00b4a0 tx=0x7fe99c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.856+0000 7fe9a57fa700 1 -- 172.21.15.89:0/4208956155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c011830 con 0x7fe9a806c000 2023-11-24T06:20:19.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.857+0000 7fe9a57fa700 1 -- 172.21.15.89:0/4208956155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c011990 con 0x7fe9a806c000 2023-11-24T06:20:19.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.857+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a8154ed0 con 0x7fe9a806c000 2023-11-24T06:20:19.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.857+0000 7fe9a57fa700 1 -- 172.21.15.89:0/4208956155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c011d40 con 0x7fe9a806c000 2023-11-24T06:20:19.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.857+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a8155490 con 0x7fe9a806c000 2023-11-24T06:20:19.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.858+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a8155800 con 0x7fe9a806c000 2023-11-24T06:20:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.858+0000 7fe9a57fa700 1 -- 172.21.15.89:0/4208956155 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe99c01a400 con 0x7fe9a806c000 2023-11-24T06:20:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.858+0000 7fe9a57fa700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98c06aed0 0x7fe98c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.858+0000 7fe9a57fa700 1 -- 172.21.15.89:0/4208956155 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe99c089240 con 0x7fe9a806c000 2023-11-24T06:20:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.858+0000 7fe9a7fff700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98c06aed0 0x7fe98c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.859+0000 7fe9a7fff700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98c06aed0 0x7fe98c06d390 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8072a90 tx=0x7fe994008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:19.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:19.863+0000 7fe9a57fa700 1 -- 172.21.15.89:0/4208956155 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe99c054650 con 0x7fe9a806c000 2023-11-24T06:20:20.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.055+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe9a80753f0 con 0x7fe9a806c000 2023-11-24T06:20:20.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.055+0000 7fe9a57fa700 1 -- 172.21.15.89:0/4208956155 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v49) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe99c057c70 con 0x7fe9a806c000 2023-11-24T06:20:20.058 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:20.058 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-11-24T06:20:20.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:20.058 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2023-11-24T06:20:20.058 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2023-11-24T06:20:20.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:20.058 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-11-24T06:20:20.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98c06aed0 msgr2=0x7fe98c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98c06aed0 0x7fe98c06d390 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8072a90 tx=0x7fe994008030 comp rx=0 tx=0).stop 2023-11-24T06:20:20.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a806c000 msgr2=0x7fe9a814a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a806c000 0x7fe9a814a360 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00b4a0 tx=0x7fe99c00b860 comp rx=0 tx=0).stop 2023-11-24T06:20:20.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 shutdown_connections 2023-11-24T06:20:20.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98c06aed0 0x7fe98c06d390 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a815e080 0x7fe9a80b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9a806c000 0x7fe9a814a360 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 --2- 172.21.15.89:0/4208956155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9a80af4d0 0x7fe9a80b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 >> 172.21.15.89:0/4208956155 conn(0x7fe9a8071b90 msgr2=0x7fe9a806efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:20.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 shutdown_connections 2023-11-24T06:20:20.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.058+0000 7fe9aee36700 1 -- 172.21.15.89:0/4208956155 wait complete. 2023-11-24T06:20:20.071 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T06:20:20.072 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T06:20:20.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.194+0000 7fdd2a592700 1 Processor -- start 2023-11-24T06:20:20.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd2a592700 1 -- start start 2023-11-24T06:20:20.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd2a592700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240d1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd2a592700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdd240af3b0 0x7fdd240af800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd2a592700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd240d01c0 con 0x7fdd240cf040 2023-11-24T06:20:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd2a592700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd240d2470 con 0x7fdd240af3b0 2023-11-24T06:20:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd2a592700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24070910 con 0x7fdd240d1ab0 2023-11-24T06:20:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd23fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdd240cf040 0x7fdd2406fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41308/0 (socket says 172.21.15.89:41308) 2023-11-24T06:20:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd23fff700 1 -- 172.21.15.89:0/4151523065 learned_addr learned my addr 172.21.15.89:0/4151523065 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.195+0000 7fdd28b2f700 1 --2- 172.21.15.89:0/4151523065 >> v2:172.21.15.89:3300/0 conn(0x7fdd240af3b0 0x7fdd240af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd237fe700 1 --2- 172.21.15.89:0/4151523065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240d1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd22ffd700 1 -- 172.21.15.89:0/4151523065 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454453026 0 0) 0x7fdd240d01c0 con 0x7fdd240cf040 2023-11-24T06:20:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd22ffd700 1 -- 172.21.15.89:0/4151523065 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd0c0035b0 con 0x7fdd240cf040 2023-11-24T06:20:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd28b2f700 1 -- 172.21.15.89:0/4151523065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 msgr2=0x7fdd240d1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd28b2f700 1 --2- 172.21.15.89:0/4151523065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240d1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd28b2f700 1 -- 172.21.15.89:0/4151523065 >> v1:172.21.15.89:6789/0 conn(0x7fdd240cf040 legacy=0x7fdd2406fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd28b2f700 1 -- 172.21.15.89:0/4151523065 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24070bf0 con 0x7fdd240af3b0 2023-11-24T06:20:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd28b2f700 1 --2- 172.21.15.89:0/4151523065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240af3b0 0x7fdd240af800 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009d90 tx=0x7fdd1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9773df8e14855a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd22ffd700 1 -- 172.21.15.89:0/4151523065 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00e070 con 0x7fdd240af3b0 2023-11-24T06:20:20.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd22ffd700 1 -- 172.21.15.89:0/4151523065 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c00bb20 con 0x7fdd240af3b0 2023-11-24T06:20:20.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd22ffd700 1 -- 172.21.15.89:0/4151523065 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c012430 con 0x7fdd240af3b0 2023-11-24T06:20:20.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd2a592700 1 -- 172.21.15.89:0/4151523065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240af3b0 msgr2=0x7fdd240af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.196+0000 7fdd2a592700 1 --2- 172.21.15.89:0/4151523065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240af3b0 0x7fdd240af800 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009d90 tx=0x7fdd1c00b330 comp rx=0 tx=0).stop 2023-11-24T06:20:20.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.197+0000 7fdd2a592700 1 -- 172.21.15.89:0/4151523065 shutdown_connections 2023-11-24T06:20:20.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.197+0000 7fdd2a592700 1 --2- 172.21.15.89:0/4151523065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240af3b0 0x7fdd240af800 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.197+0000 7fdd2a592700 1 --2- 172.21.15.89:0/4151523065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240d1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.197+0000 7fdd2a592700 1 -- 172.21.15.89:0/4151523065 >> 172.21.15.89:0/4151523065 conn(0x7fdd2409aa00 msgr2=0x7fdd2409ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:20.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.197+0000 7fdd2a592700 1 -- 172.21.15.89:0/4151523065 shutdown_connections 2023-11-24T06:20:20.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.197+0000 7fdd2a592700 1 -- 172.21.15.89:0/4151523065 wait complete. 2023-11-24T06:20:20.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.198+0000 7fdd2a592700 1 Processor -- start 2023-11-24T06:20:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.198+0000 7fdd2a592700 1 -- start start 2023-11-24T06:20:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd2a592700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd240af3b0 0x7fdd24148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd2a592700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240cf040 0x7fdd241f10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd2a592700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd2a592700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24146000 con 0x7fdd240cf040 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd2a592700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240cb250 con 0x7fdd240af3b0 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd2a592700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240cb530 con 0x7fdd240d1ab0 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240cf040 0x7fdd241f10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240cf040 0x7fdd241f10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37812/0 (socket says 172.21.15.89:37812) 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 -- 172.21.15.89:0/133515513 learned_addr learned my addr 172.21.15.89:0/133515513 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd23fff700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd240af3b0 0x7fdd24148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd28b2f700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 -- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 msgr2=0x7fdd240ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 -- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd240af3b0 msgr2=0x7fdd24148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd240af3b0 0x7fdd24148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 -- 172.21.15.89:0/133515513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c009a70 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd28b2f700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd237fe700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240cf040 0x7fdd241f10c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e810 tx=0x7fdd1800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd20ff9700 1 -- 172.21.15.89:0/133515513 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18013610 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd20ff9700 1 -- 172.21.15.89:0/133515513 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd18013770 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd240cb810 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd20ff9700 1 -- 172.21.15.89:0/133515513 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18013b20 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.199+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd240caef0 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.200+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd24067f60 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.201+0000 7fdd20ff9700 1 -- 172.21.15.89:0/133515513 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdd18013c80 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.201+0000 7fdd20ff9700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdd1006aed0 0x7fdd1006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.201+0000 7fdd20ff9700 1 -- 172.21.15.89:0/133515513 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdd18089340 con 0x7fdd240cf040 2023-11-24T06:20:20.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.201+0000 7fdd23fff700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdd1006aed0 0x7fdd1006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.202+0000 7fdd23fff700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdd1006aed0 0x7fdd1006d390 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdd140011e0 tx=0x7fdd14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:20.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.205+0000 7fdd20ff9700 1 -- 172.21.15.89:0/133515513 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd180547d0 con 0x7fdd240cf040 2023-11-24T06:20:20.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.395+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdd24146350 con 0x7fdd240cf040 2023-11-24T06:20:20.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.614+0000 7fdd20ff9700 1 -- 172.21.15.89:0/133515513 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v50) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdd18057df0 con 0x7fdd240cf040 2023-11-24T06:20:20.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdd1006aed0 msgr2=0x7fdd1006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdd1006aed0 0x7fdd1006d390 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdd140011e0 tx=0x7fdd14009040 comp rx=0 tx=0).stop 2023-11-24T06:20:20.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240cf040 msgr2=0x7fdd241f10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240cf040 0x7fdd241f10c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e810 tx=0x7fdd1800ebd0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 shutdown_connections 2023-11-24T06:20:20.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd240d1ab0 0x7fdd240ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdd1006aed0 0x7fdd1006d390 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd240af3b0 0x7fdd24148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 --2- 172.21.15.89:0/133515513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240cf040 0x7fdd241f10c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 >> 172.21.15.89:0/133515513 conn(0x7fdd2409aa00 msgr2=0x7fdd240b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:20.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 shutdown_connections 2023-11-24T06:20:20.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.617+0000 7fdd2a592700 1 -- 172.21.15.89:0/133515513 wait complete. 2023-11-24T06:20:20.624 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2023-11-24T06:20:20.634 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T06:20:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.760+0000 7f4d65dd1700 1 Processor -- start 2023-11-24T06:20:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.760+0000 7f4d65dd1700 1 -- start start 2023-11-24T06:20:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.761+0000 7f4d65dd1700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4d600c5810 0x7f4d60165250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.761+0000 7f4d65dd1700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d60155010 con 0x7f4d600d0850 2023-11-24T06:20:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.761+0000 7f4d65dd1700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6006d4a0 con 0x7f4d6006ee90 2023-11-24T06:20:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.761+0000 7f4d65dd1700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d600d0c30 con 0x7f4d600c5810 2023-11-24T06:20:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.761+0000 7f4d5f7fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4d600d0850 0x7f4d6015b550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41322/0 (socket says 172.21.15.89:41322) 2023-11-24T06:20:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.761+0000 7f4d5f7fe700 1 -- 172.21.15.89:0/330371283 learned_addr learned my addr 172.21.15.89:0/330371283 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.761+0000 7f4d5ffff700 1 --2- 172.21.15.89:0/330371283 >> v2:172.21.15.89:3300/0 conn(0x7f4d600c5810 0x7f4d60165250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.762+0000 7f4d5ffff700 1 -- 172.21.15.89:0/330371283 >> v1:172.21.15.123:6789/0 conn(0x7f4d6006ee90 legacy=0x7f4d600c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.762+0000 7f4d5ffff700 1 -- 172.21.15.89:0/330371283 >> v1:172.21.15.89:6789/0 conn(0x7f4d600d0850 legacy=0x7f4d6015b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.762+0000 7f4d5ffff700 1 -- 172.21.15.89:0/330371283 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6006d700 con 0x7f4d600c5810 2023-11-24T06:20:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.762+0000 7f4d5ffff700 1 --2- 172.21.15.89:0/330371283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600c5810 0x7f4d60165250 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009d90 tx=0x7f4d5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d96087653b47c59 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.762+0000 7f4d5e7fc700 1 -- 172.21.15.89:0/330371283 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5800bbd0 con 0x7f4d600c5810 2023-11-24T06:20:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.762+0000 7f4d5e7fc700 1 -- 172.21.15.89:0/330371283 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5800bd30 con 0x7f4d600c5810 2023-11-24T06:20:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.762+0000 7f4d5e7fc700 1 -- 172.21.15.89:0/330371283 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5800bea0 con 0x7f4d600c5810 2023-11-24T06:20:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.762+0000 7f4d65dd1700 1 -- 172.21.15.89:0/330371283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600c5810 msgr2=0x7f4d60165250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.763+0000 7f4d65dd1700 1 --2- 172.21.15.89:0/330371283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600c5810 0x7f4d60165250 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009d90 tx=0x7f4d5800b330 comp rx=0 tx=0).stop 2023-11-24T06:20:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.763+0000 7f4d65dd1700 1 -- 172.21.15.89:0/330371283 shutdown_connections 2023-11-24T06:20:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.763+0000 7f4d65dd1700 1 --2- 172.21.15.89:0/330371283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600c5810 0x7f4d60165250 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.763+0000 7f4d65dd1700 1 -- 172.21.15.89:0/330371283 >> 172.21.15.89:0/330371283 conn(0x7f4d600726e0 msgr2=0x7f4d60072af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.763+0000 7f4d65dd1700 1 -- 172.21.15.89:0/330371283 shutdown_connections 2023-11-24T06:20:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.763+0000 7f4d65dd1700 1 -- 172.21.15.89:0/330371283 wait complete. 2023-11-24T06:20:20.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.764+0000 7f4d65dd1700 1 Processor -- start 2023-11-24T06:20:20.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.764+0000 7f4d65dd1700 1 -- start start 2023-11-24T06:20:20.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d65dd1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d6006ee90 0x7f4d601f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d65dd1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d600c5810 0x7f4d601596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d65dd1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600d0850 0x7f4d6015d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d65dd1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d601f5cc0 con 0x7f4d600d0850 2023-11-24T06:20:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d65dd1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60159be0 con 0x7f4d600c5810 2023-11-24T06:20:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d65dd1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6009a5b0 con 0x7f4d6006ee90 2023-11-24T06:20:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600d0850 0x7f4d6015d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600d0850 0x7f4d6015d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37828/0 (socket says 172.21.15.89:37828) 2023-11-24T06:20:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 -- 172.21.15.89:0/2580351015 learned_addr learned my addr 172.21.15.89:0/2580351015 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5effd700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d600c5810 0x7f4d601596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 -- 172.21.15.89:0/2580351015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d6006ee90 msgr2=0x7f4d601f5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d6006ee90 0x7f4d601f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 -- 172.21.15.89:0/2580351015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d600c5810 msgr2=0x7f4d601596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:20.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d600c5810 0x7f4d601596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:20.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 -- 172.21.15.89:0/2580351015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d58009a70 con 0x7f4d600d0850 2023-11-24T06:20:20.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d5ffff700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600d0850 0x7f4d6015d090 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4d6006d630 tx=0x7f4d58004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:20.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d47fff700 1 -- 172.21.15.89:0/2580351015 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5800bea0 con 0x7f4d600d0850 2023-11-24T06:20:20.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d47fff700 1 -- 172.21.15.89:0/2580351015 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d58004040 con 0x7f4d600d0850 2023-11-24T06:20:20.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6009a890 con 0x7f4d600d0850 2023-11-24T06:20:20.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d47fff700 1 -- 172.21.15.89:0/2580351015 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5800bbd0 con 0x7f4d600d0850 2023-11-24T06:20:20.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.765+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6009ae50 con 0x7f4d600d0850 2023-11-24T06:20:20.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.766+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d600c6120 con 0x7f4d600d0850 2023-11-24T06:20:20.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.767+0000 7f4d47fff700 1 -- 172.21.15.89:0/2580351015 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4d580043f0 con 0x7f4d600d0850 2023-11-24T06:20:20.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.767+0000 7f4d47fff700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4d4c06aed0 0x7f4d4c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:20.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.767+0000 7f4d47fff700 1 -- 172.21.15.89:0/2580351015 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f4d5808b080 con 0x7f4d600d0850 2023-11-24T06:20:20.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.767+0000 7f4d5f7fe700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4d4c06aed0 0x7f4d4c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:20.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.768+0000 7f4d5f7fe700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4d4c06aed0 0x7f4d4c06d390 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4d540015d0 tx=0x7f4d54007ad0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:20.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.772+0000 7f4d47fff700 1 -- 172.21.15.89:0/2580351015 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d58059e30 con 0x7f4d600d0850 2023-11-24T06:20:20.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:20.960+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4d6009e300 con 0x7f4d600d0850 2023-11-24T06:20:21.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.616+0000 7f4d47fff700 1 -- 172.21.15.89:0/2580351015 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v51) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4d580599c0 con 0x7f4d600d0850 2023-11-24T06:20:21.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.620+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4d4c06aed0 msgr2=0x7f4d4c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:21.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.620+0000 7f4d65dd1700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4d4c06aed0 0x7f4d4c06d390 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4d540015d0 tx=0x7f4d54007ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.620+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600d0850 msgr2=0x7f4d6015d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:21.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.620+0000 7f4d65dd1700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600d0850 0x7f4d6015d090 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4d6006d630 tx=0x7f4d58004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.621+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 shutdown_connections 2023-11-24T06:20:21.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.621+0000 7f4d65dd1700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d600d0850 0x7f4d6015d090 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.621+0000 7f4d65dd1700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d6006ee90 0x7f4d601f5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.621+0000 7f4d65dd1700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4d4c06aed0 0x7f4d4c06d390 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.621+0000 7f4d65dd1700 1 --2- 172.21.15.89:0/2580351015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d600c5810 0x7f4d601596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.621+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 >> 172.21.15.89:0/2580351015 conn(0x7f4d600726e0 msgr2=0x7f4d60153620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:21.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.625+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 shutdown_connections 2023-11-24T06:20:21.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.625+0000 7f4d65dd1700 1 -- 172.21.15.89:0/2580351015 wait complete. 2023-11-24T06:20:21.639 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2023-11-24T06:20:21.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T06:20:21.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.779+0000 7fe539377700 1 Processor -- start 2023-11-24T06:20:21.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.779+0000 7fe539377700 1 -- start start 2023-11-24T06:20:21.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.779+0000 7fe539377700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe5340d0890 0x7fe5340d3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:21.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.779+0000 7fe539377700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5340cefb0 0x7fe5340b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.779+0000 7fe539377700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5340b3740 con 0x7fe5340d0c70 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.779+0000 7fe539377700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5340cf850 con 0x7fe5340cefb0 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.779+0000 7fe539377700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5340d2b30 con 0x7fe5340d0890 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.779+0000 7fe5327fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5340cefb0 0x7fe5340b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe5327fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5340cefb0 0x7fe5340b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37832/0 (socket says 172.21.15.89:37832) 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe5327fc700 1 -- 172.21.15.89:0/4184546963 learned_addr learned my addr 172.21.15.89:0/4184546963 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe532ffd700 1 --2- 172.21.15.89:0/4184546963 >> v2:172.21.15.123:3300/0 conn(0x7fe5340d0890 0x7fe5340d3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe5327fc700 1 -- 172.21.15.89:0/4184546963 >> v2:172.21.15.123:3300/0 conn(0x7fe5340d0890 msgr2=0x7fe5340d3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe5327fc700 1 --2- 172.21.15.89:0/4184546963 >> v2:172.21.15.123:3300/0 conn(0x7fe5340d0890 0x7fe5340d3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe5327fc700 1 -- 172.21.15.89:0/4184546963 >> v1:172.21.15.89:6789/0 conn(0x7fe5340d0c70 legacy=0x7fe5340b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe5327fc700 1 -- 172.21.15.89:0/4184546963 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5340cf570 con 0x7fe5340cefb0 2023-11-24T06:20:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe531ffb700 1 -- 172.21.15.89:0/4184546963 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317029397 0 0) 0x7fe5340b3740 con 0x7fe5340d0c70 2023-11-24T06:20:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe5327fc700 1 --2- 172.21.15.89:0/4184546963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340cefb0 0x7fe5340b3200 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe5340b6aa0 tx=0x7fe524009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76edda26976484dd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe531ffb700 1 -- 172.21.15.89:0/4184546963 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524011830 con 0x7fe5340cefb0 2023-11-24T06:20:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe531ffb700 1 -- 172.21.15.89:0/4184546963 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe524011990 con 0x7fe5340cefb0 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.780+0000 7fe531ffb700 1 -- 172.21.15.89:0/4184546963 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524011d20 con 0x7fe5340cefb0 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.781+0000 7fe539377700 1 -- 172.21.15.89:0/4184546963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340cefb0 msgr2=0x7fe5340b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.781+0000 7fe539377700 1 --2- 172.21.15.89:0/4184546963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340cefb0 0x7fe5340b3200 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe5340b6aa0 tx=0x7fe524009bd0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.781+0000 7fe539377700 1 -- 172.21.15.89:0/4184546963 shutdown_connections 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.781+0000 7fe539377700 1 --2- 172.21.15.89:0/4184546963 >> v2:172.21.15.123:3300/0 conn(0x7fe5340d0890 0x7fe5340d3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.781+0000 7fe539377700 1 --2- 172.21.15.89:0/4184546963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340cefb0 0x7fe5340b3200 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.781+0000 7fe539377700 1 -- 172.21.15.89:0/4184546963 >> 172.21.15.89:0/4184546963 conn(0x7fe53406b410 msgr2=0x7fe53406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.781+0000 7fe539377700 1 -- 172.21.15.89:0/4184546963 shutdown_connections 2023-11-24T06:20:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.781+0000 7fe539377700 1 -- 172.21.15.89:0/4184546963 wait complete. 2023-11-24T06:20:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.782+0000 7fe539377700 1 Processor -- start 2023-11-24T06:20:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.782+0000 7fe539377700 1 -- start start 2023-11-24T06:20:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.783+0000 7fe539377700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5340cefb0 0x7fe5341f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.783+0000 7fe539377700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5340d0890 0x7fe5340c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.783+0000 7fe539377700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340d0c70 0x7fe5341f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.783+0000 7fe539377700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340c9610 con 0x7fe5340d0c70 2023-11-24T06:20:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.783+0000 7fe539377700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340c98f0 con 0x7fe5340d0890 2023-11-24T06:20:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.783+0000 7fe539377700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340c9bd0 con 0x7fe5340cefb0 2023-11-24T06:20:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.783+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340d0c70 0x7fe5341f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340d0c70 0x7fe5341f0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37842/0 (socket says 172.21.15.89:37842) 2023-11-24T06:20:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5337fe700 1 -- 172.21.15.89:0/4045393091 learned_addr learned my addr 172.21.15.89:0/4045393091 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe532ffd700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5340cefb0 0x7fe5341f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5327fc700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5340d0890 0x7fe5340c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5337fe700 1 -- 172.21.15.89:0/4045393091 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5340cefb0 msgr2=0x7fe5341f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5337fe700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5340cefb0 0x7fe5341f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5337fe700 1 -- 172.21.15.89:0/4045393091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5340d0890 msgr2=0x7fe5340c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5337fe700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5340d0890 0x7fe5340c8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5337fe700 1 -- 172.21.15.89:0/4045393091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524009550 con 0x7fe5340d0c70 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe5337fe700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340d0c70 0x7fe5341f0e60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe52c001560 tx=0x7fe52c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe51b7fe700 1 -- 172.21.15.89:0/4045393091 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c003e30 con 0x7fe5340d0c70 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe51b7fe700 1 -- 172.21.15.89:0/4045393091 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c005070 con 0x7fe5340d0c70 2023-11-24T06:20:21.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5340c9eb0 con 0x7fe5340d0c70 2023-11-24T06:20:21.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe51b7fe700 1 -- 172.21.15.89:0/4045393091 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c005420 con 0x7fe5340d0c70 2023-11-24T06:20:21.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.784+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5340c9270 con 0x7fe5340d0c70 2023-11-24T06:20:21.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.785+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe534068440 con 0x7fe5340d0c70 2023-11-24T06:20:21.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.786+0000 7fe51b7fe700 1 -- 172.21.15.89:0/4045393091 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe52c005580 con 0x7fe5340d0c70 2023-11-24T06:20:21.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.786+0000 7fe51b7fe700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe52006aed0 0x7fe52006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:21.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.786+0000 7fe51b7fe700 1 -- 172.21.15.89:0/4045393091 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe52c088ba0 con 0x7fe5340d0c70 2023-11-24T06:20:21.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.786+0000 7fe532ffd700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe52006aed0 0x7fe52006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:21.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.787+0000 7fe532ffd700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe52006aed0 0x7fe52006d390 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe528009be0 tx=0x7fe528009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:21.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:21.791+0000 7fe51b7fe700 1 -- 172.21.15.89:0/4045393091 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe52c053cb0 con 0x7fe5340d0c70 2023-11-24T06:20:22.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.025+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe534068fb0 con 0x7fe5340d0c70 2023-11-24T06:20:22.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.644+0000 7fe51b7fe700 1 -- 172.21.15.89:0/4045393091 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v27) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe52c017070 con 0x7fe5340d0c70 2023-11-24T06:20:22.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.651+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe52006aed0 msgr2=0x7fe52006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:22.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.651+0000 7fe539377700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe52006aed0 0x7fe52006d390 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe528009be0 tx=0x7fe528009210 comp rx=0 tx=0).stop 2023-11-24T06:20:22.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.651+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340d0c70 msgr2=0x7fe5341f0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:22.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.651+0000 7fe539377700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340d0c70 0x7fe5341f0e60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe52c001560 tx=0x7fe52c00cb50 comp rx=0 tx=0).stop 2023-11-24T06:20:22.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.653+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 shutdown_connections 2023-11-24T06:20:22.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.653+0000 7fe539377700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5340d0c70 0x7fe5341f0e60 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.653+0000 7fe539377700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5340cefb0 0x7fe5341f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.653+0000 7fe539377700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe52006aed0 0x7fe52006d390 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.653+0000 7fe539377700 1 --2- 172.21.15.89:0/4045393091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5340d0890 0x7fe5340c8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.653+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 >> 172.21.15.89:0/4045393091 conn(0x7fe53406b410 msgr2=0x7fe5340b5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:22.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.654+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 shutdown_connections 2023-11-24T06:20:22.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.654+0000 7fe539377700 1 -- 172.21.15.89:0/4045393091 wait complete. 2023-11-24T06:20:22.666 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 8 and data pool 9 2023-11-24T06:20:22.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.799+0000 7fe97036d700 1 Processor -- start 2023-11-24T06:20:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.799+0000 7fe97036d700 1 -- start start 2023-11-24T06:20:22.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.800+0000 7fe97036d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 0x7fe9680b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:22.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.800+0000 7fe97036d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680d1190 con 0x7fe96815e1e0 2023-11-24T06:20:22.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.800+0000 7fe97036d700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680d20b0 con 0x7fe9680d0870 2023-11-24T06:20:22.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.800+0000 7fe97036d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680d3150 con 0x7fe9680d0d00 2023-11-24T06:20:22.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96e109700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe96815e1e0 0x7fe9680d34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41362/0 (socket says 172.21.15.89:41362) 2023-11-24T06:20:22.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96e109700 1 -- 172.21.15.89:0/2013655872 learned_addr learned my addr 172.21.15.89:0/2013655872 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96e90a700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 0x7fe9680b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252200412 0 0) 0x7fe9680d1190 con 0x7fe96815e1e0 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9540035b0 con 0x7fe96815e1e0 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653488776 0 0) 0x7fe9680d20b0 con 0x7fe9680d0870 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9680d1190 con 0x7fe9680d0870 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938234741 0 0) 0x7fe9540035b0 con 0x7fe96815e1e0 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 msgr2=0x7fe9680b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 0x7fe9680b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 >> v1:172.21.15.123:6789/0 conn(0x7fe9680d0870 legacy=0x7fe9680b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680d2390 con 0x7fe96815e1e0 2023-11-24T06:20:22.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fe95c001670 con 0x7fe96815e1e0 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96e90a700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 0x7fe9680b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 >> v1:172.21.15.89:6789/0 conn(0x7fe96815e1e0 legacy=0x7fe9680d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 0x7fe954003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954004100 0x7fe954004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe954004d90 0x7fe954005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680d3150 con 0x7fe954004100 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95400bac0 con 0x7fe954004d90 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95400bc80 con 0x7fe9680d0d00 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d908700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954004100 0x7fe954004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96e109700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 0x7fe954003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96e90a700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe954004d90 0x7fe954005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d908700 1 -- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 msgr2=0x7fe954003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d908700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9680d0d00 0x7fe954003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d908700 1 -- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe954004d90 msgr2=0x7fe954005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.801+0000 7fe96d908700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe954004d90 0x7fe954005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe96d908700 1 -- 172.21.15.89:0/2013655872 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe960009a70 con 0x7fe954004100 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe96d908700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954004100 0x7fe954004850 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe964001980 tx=0x7fe954005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f022e67946c4c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe964001470 con 0x7fe954004100 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9640052d0 con 0x7fe954004100 2023-11-24T06:20:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe96d107700 1 -- 172.21.15.89:0/2013655872 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe964003680 con 0x7fe954004100 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe97036d700 1 -- 172.21.15.89:0/2013655872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954004100 msgr2=0x7fe954004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe97036d700 1 --2- 172.21.15.89:0/2013655872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe954004100 0x7fe954004850 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe964001980 tx=0x7fe954005880 comp rx=0 tx=0).stop 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe96e90a700 1 -- 172.21.15.89:0/2013655872 reap_dead start 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe97036d700 1 -- 172.21.15.89:0/2013655872 shutdown_connections 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe97036d700 1 -- 172.21.15.89:0/2013655872 >> 172.21.15.89:0/2013655872 conn(0x7fe9680b5b50 msgr2=0x7fe9680b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe97036d700 1 -- 172.21.15.89:0/2013655872 shutdown_connections 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.802+0000 7fe97036d700 1 -- 172.21.15.89:0/2013655872 wait complete. 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe97036d700 1 Processor -- start 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe97036d700 1 -- start start 2023-11-24T06:20:22.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe97036d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe954004100 0x7fe9680ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe97036d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9680cc100 0x7fe968149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe97036d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe968149ac0 0x7fe9680ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe97036d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680cbc70 con 0x7fe9680cc100 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe97036d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680cb060 con 0x7fe968149ac0 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe97036d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680cb340 con 0x7fe954004100 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe96d908700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9680cc100 0x7fe968149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe96d908700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9680cc100 0x7fe968149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37872/0 (socket says 172.21.15.89:37872) 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe96d908700 1 -- 172.21.15.89:0/2206886852 learned_addr learned my addr 172.21.15.89:0/2206886852 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.804+0000 7fe96e90a700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe968149ac0 0x7fe9680ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe96e109700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe954004100 0x7fe9680ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe96d908700 1 -- 172.21.15.89:0/2206886852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe954004100 msgr2=0x7fe9680ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe96d908700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe954004100 0x7fe9680ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe96d908700 1 -- 172.21.15.89:0/2206886852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe968149ac0 msgr2=0x7fe9680ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe96d908700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe968149ac0 0x7fe9680ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe96d908700 1 -- 172.21.15.89:0/2206886852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680cb620 con 0x7fe9680cc100 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe96d908700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9680cc100 0x7fe968149580 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe964006a90 tx=0x7fe964001310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe95affd700 1 -- 172.21.15.89:0/2206886852 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe964001110 con 0x7fe9680cc100 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe95affd700 1 -- 172.21.15.89:0/2206886852 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe964001410 con 0x7fe9680cc100 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9680cbc70 con 0x7fe9680cc100 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe95affd700 1 -- 172.21.15.89:0/2206886852 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe964007280 con 0x7fe9680cc100 2023-11-24T06:20:22.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.805+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe968160fc0 con 0x7fe9680cc100 2023-11-24T06:20:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.806+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9680cbe00 con 0x7fe9680cc100 2023-11-24T06:20:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.806+0000 7fe95affd700 1 -- 172.21.15.89:0/2206886852 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe9640073e0 con 0x7fe9680cc100 2023-11-24T06:20:22.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.807+0000 7fe95affd700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe94c06aed0 0x7fe94c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:22.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.807+0000 7fe95affd700 1 -- 172.21.15.89:0/2206886852 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe96408a4d0 con 0x7fe9680cc100 2023-11-24T06:20:22.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.807+0000 7fe96e109700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe94c06aed0 0x7fe94c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:22.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.807+0000 7fe96e109700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe94c06aed0 0x7fe94c06d390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe95c001c20 tx=0x7fe95c001e00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:22.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:22.811+0000 7fe95affd700 1 -- 172.21.15.89:0/2206886852 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9640557e0 con 0x7fe9680cc100 2023-11-24T06:20:23.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.002+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9680751e0 con 0x7fe9680cc100 2023-11-24T06:20:23.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.004+0000 7fe95affd700 1 -- 172.21.15.89:0/2206886852 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16542 (secure 0 0 0) 0x7fe964058e00 con 0x7fe9680cc100 2023-11-24T06:20:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:23.006 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":52,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:22.026555+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:20.413276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:20.978712+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe94c06aed0 msgr2=0x7fe94c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe94c06aed0 0x7fe94c06d390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe95c001c20 tx=0x7fe95c001e00 comp rx=0 tx=0).stop 2023-11-24T06:20:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9680cc100 msgr2=0x7fe968149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9680cc100 0x7fe968149580 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe964006a90 tx=0x7fe964001310 comp rx=0 tx=0).stop 2023-11-24T06:20:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 shutdown_connections 2023-11-24T06:20:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe94c06aed0 0x7fe94c06d390 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe968149ac0 0x7fe9680ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe954004100 0x7fe9680ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 --2- 172.21.15.89:0/2206886852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9680cc100 0x7fe968149580 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 >> 172.21.15.89:0/2206886852 conn(0x7fe9680b5b50 msgr2=0x7fe968153270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 shutdown_connections 2023-11-24T06:20:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.006+0000 7fe97036d700 1 -- 172.21.15.89:0/2206886852 wait complete. 2023-11-24T06:20:23.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:23.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.138+0000 7f05f4fb6700 1 Processor -- start 2023-11-24T06:20:23.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.138+0000 7f05f4fb6700 1 -- start start 2023-11-24T06:20:23.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05f4fb6700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05f0152cc0 0x7f05f006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05f4fb6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00d0630 0x7f05f006c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05f4fb6700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f015c4b0 con 0x7f05f00b49a0 2023-11-24T06:20:23.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05f4fb6700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f015c790 con 0x7f05f0152cc0 2023-11-24T06:20:23.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05f4fb6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f015ca70 con 0x7f05f00d0630 2023-11-24T06:20:23.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ee59c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05f0152cc0 0x7f05f006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ee59c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05f0152cc0 0x7f05f006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37878/0 (socket says 172.21.15.89:37878) 2023-11-24T06:20:23.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ee59c700 1 -- 172.21.15.89:0/3612918672 learned_addr learned my addr 172.21.15.89:0/3612918672 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:23.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05edd9b700 1 --1- 172.21.15.89:0/3612918672 >> v1:172.21.15.89:6789/0 conn(0x7f05f00b49a0 0x7f05f00cff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41382/0 (socket says 172.21.15.89:41382) 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05eed9d700 1 --2- 172.21.15.89:0/3612918672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00d0630 0x7f05f006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ee59c700 1 -- 172.21.15.89:0/3612918672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00d0630 msgr2=0x7f05f006c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ee59c700 1 --2- 172.21.15.89:0/3612918672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00d0630 0x7f05f006c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ee59c700 1 -- 172.21.15.89:0/3612918672 >> v1:172.21.15.89:6789/0 conn(0x7f05f00b49a0 legacy=0x7f05f00cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ee59c700 1 -- 172.21.15.89:0/3612918672 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f015cd50 con 0x7f05f0152cc0 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ee59c700 1 --2- 172.21.15.89:0/3612918672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f006c000 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f05e00098f0 tx=0x7f05e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8089d0f8adc7e0b9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.139+0000 7f05ed59a700 1 -- 172.21.15.89:0/3612918672 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e000b8a0 con 0x7f05f0152cc0 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05ed59a700 1 -- 172.21.15.89:0/3612918672 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e000ba00 con 0x7f05f0152cc0 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05ed59a700 1 -- 172.21.15.89:0/3612918672 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e000bd90 con 0x7f05f0152cc0 2023-11-24T06:20:23.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05f4fb6700 1 -- 172.21.15.89:0/3612918672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 msgr2=0x7f05f006c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/3612918672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f006c000 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f05e00098f0 tx=0x7f05e0009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:23.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05f4fb6700 1 -- 172.21.15.89:0/3612918672 shutdown_connections 2023-11-24T06:20:23.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/3612918672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00d0630 0x7f05f006c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/3612918672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f006c000 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05f4fb6700 1 -- 172.21.15.89:0/3612918672 >> 172.21.15.89:0/3612918672 conn(0x7f05f0071b90 msgr2=0x7f05f0071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:23.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05f4fb6700 1 -- 172.21.15.89:0/3612918672 shutdown_connections 2023-11-24T06:20:23.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.140+0000 7f05f4fb6700 1 -- 172.21.15.89:0/3612918672 wait complete. 2023-11-24T06:20:23.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.141+0000 7f05f4fb6700 1 Processor -- start 2023-11-24T06:20:23.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05f4fb6700 1 -- start start 2023-11-24T06:20:23.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05f4fb6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00b49a0 0x7f05f0153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05f4fb6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05f00d0630 0x7f05f014ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05f4fb6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f01477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05f4fb6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f014f2c0 con 0x7f05f0152cc0 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05f4fb6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f014f520 con 0x7f05f00d0630 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05f4fb6700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f014f800 con 0x7f05f00b49a0 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05eed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f01477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05eed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f01477f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37888/0 (socket says 172.21.15.89:37888) 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05eed9d700 1 -- 172.21.15.89:0/1037398438 learned_addr learned my addr 172.21.15.89:0/1037398438 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05ee59c700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00b49a0 0x7f05f0153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.142+0000 7f05edd9b700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05f00d0630 0x7f05f014ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05eed9d700 1 -- 172.21.15.89:0/1037398438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00b49a0 msgr2=0x7f05f0153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05eed9d700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00b49a0 0x7f05f0153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05eed9d700 1 -- 172.21.15.89:0/1037398438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05f00d0630 msgr2=0x7f05f014ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05eed9d700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05f00d0630 0x7f05f014ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05eed9d700 1 -- 172.21.15.89:0/1037398438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e0009550 con 0x7f05f0152cc0 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05eed9d700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f01477f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f05e8007e40 tx=0x7f05e800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05deffd700 1 -- 172.21.15.89:0/1037398438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e800ed30 con 0x7f05f0152cc0 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05deffd700 1 -- 172.21.15.89:0/1037398438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e8014420 con 0x7f05f0152cc0 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05deffd700 1 -- 172.21.15.89:0/1037398438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e80147d0 con 0x7f05f0152cc0 2023-11-24T06:20:23.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f014fae0 con 0x7f05f0152cc0 2023-11-24T06:20:23.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.143+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f01500a0 con 0x7f05f0152cc0 2023-11-24T06:20:23.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.144+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f00753f0 con 0x7f05f0152cc0 2023-11-24T06:20:23.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.144+0000 7f05deffd700 1 -- 172.21.15.89:0/1037398438 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f05e8014930 con 0x7f05f0152cc0 2023-11-24T06:20:23.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.145+0000 7f05deffd700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f05d006aed0 0x7f05d006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.145+0000 7f05deffd700 1 -- 172.21.15.89:0/1037398438 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05e8089bc0 con 0x7f05f0152cc0 2023-11-24T06:20:23.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.145+0000 7f05ee59c700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f05d006aed0 0x7f05d006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.145+0000 7f05ee59c700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f05d006aed0 0x7f05d006d390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f05e0014040 tx=0x7f05e0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:23.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.149+0000 7f05deffd700 1 -- 172.21.15.89:0/1037398438 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05e8054d00 con 0x7f05f0152cc0 2023-11-24T06:20:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.342+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05f0067f60 con 0x7f05f0152cc0 2023-11-24T06:20:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.343+0000 7f05deffd700 1 -- 172.21.15.89:0/1037398438 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16542 (secure 0 0 0) 0x7f05e8058320 con 0x7f05f0152cc0 2023-11-24T06:20:23.345 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:23.345 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":52,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:22.026555+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:20.413276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:20.978712+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f05d006aed0 msgr2=0x7f05d006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f05d006aed0 0x7f05d006d390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f05e0014040 tx=0x7f05e0008030 comp rx=0 tx=0).stop 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 msgr2=0x7f05f01477f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f01477f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f05e8007e40 tx=0x7f05e800e3c0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 shutdown_connections 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05f0152cc0 0x7f05f01477f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05f00b49a0 0x7f05f0153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f05d006aed0 0x7f05d006d390 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 --2- 172.21.15.89:0/1037398438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05f00d0630 0x7f05f014ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 >> 172.21.15.89:0/1037398438 conn(0x7f05f0071b90 msgr2=0x7f05f00adf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 shutdown_connections 2023-11-24T06:20:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.345+0000 7f05f4fb6700 1 -- 172.21.15.89:0/1037398438 wait complete. 2023-11-24T06:20:23.360 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T06:20:23.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.481+0000 7f5408fdc700 1 Processor -- start 2023-11-24T06:20:23.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.481+0000 7f5408fdc700 1 -- start start 2023-11-24T06:20:23.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f5408fdc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f54040aed00 0x7f54040cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f5408fdc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5404071a80 0x7f540406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f5408fdc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5404156910 con 0x7f54040afa90 2023-11-24T06:20:23.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f5408fdc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540415c760 con 0x7f54040aed00 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f5408fdc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540415ca40 con 0x7f5404071a80 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f5401d9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f54040afa90 0x7f5404071370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41398/0 (socket says 172.21.15.89:41398) 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f5401d9b700 1 -- 172.21.15.89:0/4260566594 learned_addr learned my addr 172.21.15.89:0/4260566594 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540259c700 1 --2- 172.21.15.89:0/4260566594 >> v2:172.21.15.123:3300/0 conn(0x7f54040aed00 0x7f54040cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f5402d9d700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5404071a80 0x7f540406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335728386 0 0) 0x7f5404156910 con 0x7f54040afa90 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ec0035b0 con 0x7f54040afa90 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802722947 0 0) 0x7f53ec0035b0 con 0x7f54040afa90 2023-11-24T06:20:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5404071a80 msgr2=0x7f540406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5404071a80 0x7f540406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 >> v2:172.21.15.123:3300/0 conn(0x7f54040aed00 msgr2=0x7f54040cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 --2- 172.21.15.89:0/4260566594 >> v2:172.21.15.123:3300/0 conn(0x7f54040aed00 0x7f54040cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540415cd20 con 0x7f54040afa90 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f53f8001670 con 0x7f54040afa90 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 >> v1:172.21.15.89:6789/0 conn(0x7f54040afa90 legacy=0x7f5404071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.482+0000 7f540159a700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540159a700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5404071a80 0x7f53ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540159a700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004d90 0x7f53ec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540415c760 con 0x7f53ec004d90 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540415ca40 con 0x7f53ec003a00 2023-11-24T06:20:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec00ba10 con 0x7f5404071a80 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f5402d9d700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004d90 0x7f53ec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540259c700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f5401d9b700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5404071a80 0x7f53ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f5402d9d700 1 -- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5404071a80 msgr2=0x7f53ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f5402d9d700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5404071a80 0x7f53ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f5402d9d700 1 -- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec003a00 msgr2=0x7f53ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f5402d9d700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f5402d9d700 1 -- 172.21.15.89:0/4260566594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0035b0 con 0x7f53ec004d90 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f5402d9d700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004d90 0x7f53ec005210 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009f90 tx=0x7f53ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594ca2bc876ee22f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc00ebd0 con 0x7f53ec004d90 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc00f040 con 0x7f53ec004d90 2023-11-24T06:20:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.483+0000 7f540159a700 1 -- 172.21.15.89:0/4260566594 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc014480 con 0x7f53ec004d90 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.484+0000 7f5408fdc700 1 -- 172.21.15.89:0/4260566594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004d90 msgr2=0x7f53ec005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.484+0000 7f5408fdc700 1 --2- 172.21.15.89:0/4260566594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004d90 0x7f53ec005210 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009f90 tx=0x7f53ec005880 comp rx=0 tx=0).stop 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.484+0000 7f5402d9d700 1 -- 172.21.15.89:0/4260566594 reap_dead start 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.484+0000 7f5408fdc700 1 -- 172.21.15.89:0/4260566594 shutdown_connections 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.484+0000 7f5408fdc700 1 -- 172.21.15.89:0/4260566594 >> 172.21.15.89:0/4260566594 conn(0x7f540406aff0 msgr2=0x7f540406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.484+0000 7f5408fdc700 1 -- 172.21.15.89:0/4260566594 shutdown_connections 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.484+0000 7f5408fdc700 1 -- 172.21.15.89:0/4260566594 wait complete. 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.485+0000 7f5408fdc700 1 Processor -- start 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.485+0000 7f5408fdc700 1 -- start start 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5408fdc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004d90 0x7f54040ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5408fdc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5404151400 0x7f54041418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5408fdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5404151880 0x7f5404141e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5408fdc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54041500f0 con 0x7f5404151880 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5408fdc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540414e3b0 con 0x7f5404151400 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5408fdc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540414f4e0 con 0x7f53ec004d90 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5404151880 0x7f5404141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5404151880 0x7f5404141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37908/0 (socket says 172.21.15.89:37908) 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 -- 172.21.15.89:0/3561999963 learned_addr learned my addr 172.21.15.89:0/3561999963 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5401d9b700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5404151400 0x7f54041418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f540259c700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004d90 0x7f54040ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 -- 172.21.15.89:0/3561999963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004d90 msgr2=0x7f54040ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004d90 0x7f54040ab730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 -- 172.21.15.89:0/3561999963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5404151400 msgr2=0x7f54041418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5404151400 0x7f54041418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 -- 172.21.15.89:0/3561999963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540414f7c0 con 0x7f5404151880 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.486+0000 7f5402d9d700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5404151880 0x7f5404141e00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f53ec004730 tx=0x7f53fc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.487+0000 7f53f2ffd700 1 -- 172.21.15.89:0/3561999963 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc007000 con 0x7f5404151880 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.487+0000 7f53f2ffd700 1 -- 172.21.15.89:0/3561999963 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc014d60 con 0x7f5404151880 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.487+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540414faa0 con 0x7f5404151880 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.487+0000 7f53f2ffd700 1 -- 172.21.15.89:0/3561999963 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc003c60 con 0x7f5404151880 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.487+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540414f1b0 con 0x7f5404151880 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.487+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5404068fb0 con 0x7f5404151880 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.488+0000 7f53f2ffd700 1 -- 172.21.15.89:0/3561999963 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f53fc023070 con 0x7f5404151880 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.489+0000 7f53f2ffd700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53e406aed0 0x7f53e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:23.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.489+0000 7f53f2ffd700 1 -- 172.21.15.89:0/3561999963 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53fc08acf0 con 0x7f5404151880 2023-11-24T06:20:23.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.489+0000 7f540259c700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53e406aed0 0x7f53e406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:23.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.489+0000 7f540259c700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53e406aed0 0x7f53e406d390 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009aa0 tx=0x7f53f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:23.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.492+0000 7f53f2ffd700 1 -- 172.21.15.89:0/3561999963 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53fc056030 con 0x7f5404151880 2023-11-24T06:20:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:23.729+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f54040cd600 con 0x7f5404151880 2023-11-24T06:20:24.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.658+0000 7f53f2ffd700 1 -- 172.21.15.89:0/3561999963 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7f53fc059650 con 0x7f5404151880 2023-11-24T06:20:24.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53e406aed0 msgr2=0x7f53e406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53e406aed0 0x7f53e406d390 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009aa0 tx=0x7f53f4008040 comp rx=0 tx=0).stop 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5404151880 msgr2=0x7f5404141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5404151880 0x7f5404141e00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f53ec004730 tx=0x7f53fc009750 comp rx=0 tx=0).stop 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 shutdown_connections 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5404151880 0x7f5404141e00 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004d90 0x7f54040ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53e406aed0 0x7f53e406d390 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 --2- 172.21.15.89:0/3561999963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5404151400 0x7f54041418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.660+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 >> 172.21.15.89:0/3561999963 conn(0x7f540406aff0 msgr2=0x7f54040aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.661+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 shutdown_connections 2023-11-24T06:20:24.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.661+0000 7f5408fdc700 1 -- 172.21.15.89:0/3561999963 wait complete. 2023-11-24T06:20:24.676 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T06:20:24.677 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T06:20:24.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.805+0000 7fb0ef57c700 1 Processor -- start 2023-11-24T06:20:24.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.805+0000 7fb0ef57c700 1 -- start start 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0ef57c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb0e806ecf0 0x7fb0e8165180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0ef57c700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80b2f20 con 0x7fb0e815a620 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0ef57c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e815aa00 con 0x7fb0e80cc540 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0ef57c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e80ce960 con 0x7fb0e806ecf0 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0ed318700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb0e815a620 0x7fb0e80ce150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41416/0 (socket says 172.21.15.89:41416) 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0ed318700 1 -- 172.21.15.89:0/1563778285 learned_addr learned my addr 172.21.15.89:0/1563778285 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0edb19700 1 --2- 172.21.15.89:0/1563778285 >> v2:172.21.15.123:3300/0 conn(0x7fb0e806ecf0 0x7fb0e8165180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10594730 0 0) 0x7fb0e80b2f20 con 0x7fb0e815a620 2023-11-24T06:20:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d40046c0 con 0x7fb0e815a620 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 265632950 0 0) 0x7fb0d40046c0 con 0x7fb0e815a620 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 >> v2:172.21.15.123:3300/0 conn(0x7fb0e806ecf0 msgr2=0x7fb0e8165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 --2- 172.21.15.89:0/1563778285 >> v2:172.21.15.123:3300/0 conn(0x7fb0e806ecf0 0x7fb0e8165180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 >> v1:172.21.15.123:6789/0 conn(0x7fb0e80cc540 legacy=0x7fb0e806e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e80cec40 con 0x7fb0e815a620 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fb0d8001670 con 0x7fb0e815a620 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 >> v1:172.21.15.89:6789/0 conn(0x7fb0e815a620 legacy=0x7fb0e80ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4004b10 0x7fb0d4004ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4005510 0x7fb0d40059e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d40061e0 0x7fb0d400cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80ce960 con 0x7fb0d4005510 2023-11-24T06:20:24.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d400d140 con 0x7fb0d40061e0 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.806+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d400d300 con 0x7fb0d4004b10 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0ecb17700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4005510 0x7fb0d40059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0edb19700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d40061e0 0x7fb0d400cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0ed318700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4004b10 0x7fb0d4004ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0ecb17700 1 -- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4004b10 msgr2=0x7fb0d4004ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0ecb17700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4004b10 0x7fb0d4004ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0ecb17700 1 -- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d40061e0 msgr2=0x7fb0d400cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0ecb17700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d40061e0 0x7fb0d400cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0ecb17700 1 -- 172.21.15.89:0/1563778285 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40046c0 con 0x7fb0d4005510 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0edb19700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0d40061e0 0x7fb0d400cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0edb19700 1 -- 172.21.15.89:0/1563778285 reap_dead start 2023-11-24T06:20:24.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0ecb17700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4005510 0x7fb0d40059e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40018d0 tx=0x7fb0d4006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86461d34b138a3f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e40052d0 con 0x7fb0d4005510 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4005430 con 0x7fb0d4005510 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.807+0000 7fb0dffff700 1 -- 172.21.15.89:0/1563778285 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e40038a0 con 0x7fb0d4005510 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.808+0000 7fb0ef57c700 1 -- 172.21.15.89:0/1563778285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4005510 msgr2=0x7fb0d40059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.808+0000 7fb0ef57c700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4005510 0x7fb0d40059e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40018d0 tx=0x7fb0d4006990 comp rx=0 tx=0).stop 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.808+0000 7fb0ef57c700 1 -- 172.21.15.89:0/1563778285 shutdown_connections 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.808+0000 7fb0ef57c700 1 --2- 172.21.15.89:0/1563778285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4005510 0x7fb0d40059e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.808+0000 7fb0ef57c700 1 -- 172.21.15.89:0/1563778285 >> 172.21.15.89:0/1563778285 conn(0x7fb0e80b2050 msgr2=0x7fb0e80b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.808+0000 7fb0ef57c700 1 -- 172.21.15.89:0/1563778285 shutdown_connections 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.808+0000 7fb0ef57c700 1 -- 172.21.15.89:0/1563778285 wait complete. 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.809+0000 7fb0ef57c700 1 Processor -- start 2023-11-24T06:20:24.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.809+0000 7fb0ef57c700 1 -- start start 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ef57c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4005510 0x7fb0e813c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ef57c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e8152430 0x7fb0e81e57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ef57c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0e80b2f70 0x7fb0e80b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ef57c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80b3930 con 0x7fb0e8152430 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ef57c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81e5d00 con 0x7fb0e80b2f70 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ef57c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8152ba0 con 0x7fb0d4005510 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e8152430 0x7fb0e81e57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e8152430 0x7fb0e81e57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37926/0 (socket says 172.21.15.89:37926) 2023-11-24T06:20:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 -- 172.21.15.89:0/2792903397 learned_addr learned my addr 172.21.15.89:0/2792903397 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ed318700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4005510 0x7fb0e813c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 -- 172.21.15.89:0/2792903397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4005510 msgr2=0x7fb0e813c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4005510 0x7fb0e813c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 -- 172.21.15.89:0/2792903397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0e80b2f70 msgr2=0x7fb0e80b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0e80b2f70 0x7fb0e80b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 -- 172.21.15.89:0/2792903397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e8152e80 con 0x7fb0e8152430 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ecb17700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e8152430 0x7fb0e81e57c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400c0e0 tx=0x7fb0e4001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ddffb700 1 -- 172.21.15.89:0/2792903397 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4006a60 con 0x7fb0e8152430 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ddffb700 1 -- 172.21.15.89:0/2792903397 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4006c00 con 0x7fb0e8152430 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8153160 con 0x7fb0e8152430 2023-11-24T06:20:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.810+0000 7fb0ddffb700 1 -- 172.21.15.89:0/2792903397 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4000ed0 con 0x7fb0e8152430 2023-11-24T06:20:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.811+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e8153720 con 0x7fb0e8152430 2023-11-24T06:20:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.811+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e8068d90 con 0x7fb0e8152430 2023-11-24T06:20:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.812+0000 7fb0ddffb700 1 -- 172.21.15.89:0/2792903397 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb0e4007020 con 0x7fb0e8152430 2023-11-24T06:20:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.812+0000 7fb0ddffb700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb0cc06aed0 0x7fb0cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.812+0000 7fb0ddffb700 1 -- 172.21.15.89:0/2792903397 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0e408a4e0 con 0x7fb0e8152430 2023-11-24T06:20:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.812+0000 7fb0ed318700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb0cc06aed0 0x7fb0cc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.813+0000 7fb0ed318700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb0cc06aed0 0x7fb0cc06d390 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8001050 tx=0x7fb0d8001d60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:24.817+0000 7fb0ddffb700 1 -- 172.21.15.89:0/2792903397 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0e4055620 con 0x7fb0e8152430 2023-11-24T06:20:25.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.053+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb0e81538b0 con 0x7fb0e8152430 2023-11-24T06:20:25.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.660+0000 7fb0ddffb700 1 -- 172.21.15.89:0/2792903397 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb0e405f030 con 0x7fb0e8152430 2023-11-24T06:20:25.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb0cc06aed0 msgr2=0x7fb0cc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb0cc06aed0 0x7fb0cc06d390 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8001050 tx=0x7fb0d8001d60 comp rx=0 tx=0).stop 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e8152430 msgr2=0x7fb0e81e57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e8152430 0x7fb0e81e57c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400c0e0 tx=0x7fb0e4001230 comp rx=0 tx=0).stop 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 shutdown_connections 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb0cc06aed0 0x7fb0cc06d390 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0e80b2f70 0x7fb0e80b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0d4005510 0x7fb0e813c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 --2- 172.21.15.89:0/2792903397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e8152430 0x7fb0e81e57c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.662+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 >> 172.21.15.89:0/2792903397 conn(0x7fb0e80b2050 msgr2=0x7fb0e8159090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.663+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 shutdown_connections 2023-11-24T06:20:25.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.663+0000 7fb0ef57c700 1 -- 172.21.15.89:0/2792903397 wait complete. 2023-11-24T06:20:25.679 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T06:20:25.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:25.804 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.802+0000 7fafd4920700 1 Processor -- start 2023-11-24T06:20:25.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.802+0000 7fafd4920700 1 -- start start 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.803+0000 7fafd4920700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd015bdd0 0x7fafd01604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.803+0000 7fafd4920700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fafd00b3370 0x7fafd00b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.803+0000 7fafd4920700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00b2030 con 0x7fafd00d1c70 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.803+0000 7fafd4920700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd00b2f50 con 0x7fafd00b3370 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.803+0000 7fafd4920700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00b22d0 con 0x7fafd015bdd0 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcdd9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fafd00b3370 0x7fafd00b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcdd9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fafd00b3370 0x7fafd00b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37932/0 (socket says 172.21.15.89:37932) 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcdd9b700 1 -- 172.21.15.89:0/1585595785 learned_addr learned my addr 172.21.15.89:0/1585595785 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:25.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcdd9b700 1 -- 172.21.15.89:0/1585595785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd015bdd0 msgr2=0x7fafd01604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcdd9b700 1 --2- 172.21.15.89:0/1585595785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd015bdd0 0x7fafd01604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcdd9b700 1 -- 172.21.15.89:0/1585595785 >> v1:172.21.15.123:6789/0 conn(0x7fafd00d1c70 legacy=0x7fafd00ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcdd9b700 1 -- 172.21.15.89:0/1585595785 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00b25b0 con 0x7fafd00b3370 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcdd9b700 1 --2- 172.21.15.89:0/1585595785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00b3370 0x7fafd00b1af0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fafc00098f0 tx=0x7fafc0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258ab778d5ffb1a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcd59a700 1 -- 172.21.15.89:0/1585595785 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0004230 con 0x7fafd00b3370 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcd59a700 1 -- 172.21.15.89:0/1585595785 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0004390 con 0x7fafd00b3370 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.804+0000 7fafcd59a700 1 -- 172.21.15.89:0/1585595785 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00036a0 con 0x7fafd00b3370 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.805+0000 7fafd4920700 1 -- 172.21.15.89:0/1585595785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00b3370 msgr2=0x7fafd00b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.805+0000 7fafd4920700 1 --2- 172.21.15.89:0/1585595785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00b3370 0x7fafd00b1af0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fafc00098f0 tx=0x7fafc0009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.805+0000 7fafd4920700 1 -- 172.21.15.89:0/1585595785 shutdown_connections 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.805+0000 7fafd4920700 1 --2- 172.21.15.89:0/1585595785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd015bdd0 0x7fafd01604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.805+0000 7fafd4920700 1 --2- 172.21.15.89:0/1585595785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00b3370 0x7fafd00b1af0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.805+0000 7fafd4920700 1 -- 172.21.15.89:0/1585595785 >> 172.21.15.89:0/1585595785 conn(0x7fafd00731a0 msgr2=0x7fafd009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.805+0000 7fafd4920700 1 -- 172.21.15.89:0/1585595785 shutdown_connections 2023-11-24T06:20:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.805+0000 7fafd4920700 1 -- 172.21.15.89:0/1585595785 wait complete. 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.806+0000 7fafd4920700 1 Processor -- start 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.806+0000 7fafd4920700 1 -- start start 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafd4920700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd00b3370 0x7fafd0156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafd4920700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd00d1c70 0x7fafd00cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafd4920700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd015bdd0 0x7fafd01e6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafd4920700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00c97c0 con 0x7fafd015bdd0 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafd4920700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00c9aa0 con 0x7fafd00d1c70 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafd4920700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00c9d80 con 0x7fafd00b3370 2023-11-24T06:20:25.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafced9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd015bdd0 0x7fafd01e6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafced9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd015bdd0 0x7fafd01e6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37942/0 (socket says 172.21.15.89:37942) 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafced9d700 1 -- 172.21.15.89:0/498054483 learned_addr learned my addr 172.21.15.89:0/498054483 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafcdd9b700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd00d1c70 0x7fafd00cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafce59c700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd00b3370 0x7fafd0156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafced9d700 1 -- 172.21.15.89:0/498054483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd00b3370 msgr2=0x7fafd0156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafced9d700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd00b3370 0x7fafd0156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafced9d700 1 -- 172.21.15.89:0/498054483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd00d1c70 msgr2=0x7fafd00cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafced9d700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd00d1c70 0x7fafd00cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.807+0000 7fafced9d700 1 -- 172.21.15.89:0/498054483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc0009550 con 0x7fafd015bdd0 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.808+0000 7fafced9d700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd015bdd0 0x7fafd01e6860 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fafc800cc10 tx=0x7fafc800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.808+0000 7fafc6ffd700 1 -- 172.21.15.89:0/498054483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc800ab10 con 0x7fafd015bdd0 2023-11-24T06:20:25.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.808+0000 7fafc6ffd700 1 -- 172.21.15.89:0/498054483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc800ac70 con 0x7fafd015bdd0 2023-11-24T06:20:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.808+0000 7fafc6ffd700 1 -- 172.21.15.89:0/498054483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8012420 con 0x7fafd015bdd0 2023-11-24T06:20:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.808+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd00ca060 con 0x7fafd015bdd0 2023-11-24T06:20:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.808+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd00ca620 con 0x7fafd015bdd0 2023-11-24T06:20:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.809+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd00ca7e0 con 0x7fafd015bdd0 2023-11-24T06:20:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.809+0000 7fafc6ffd700 1 -- 172.21.15.89:0/498054483 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fafc8012580 con 0x7fafd015bdd0 2023-11-24T06:20:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.809+0000 7fafc6ffd700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fafb006aed0 0x7fafb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.810+0000 7fafc6ffd700 1 -- 172.21.15.89:0/498054483 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafc8089890 con 0x7fafd015bdd0 2023-11-24T06:20:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.810+0000 7fafce59c700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fafb006aed0 0x7fafb006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.810+0000 7fafce59c700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fafb006aed0 0x7fafb006d390 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fafb8006fd0 tx=0x7fafb8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:25.814+0000 7fafc6ffd700 1 -- 172.21.15.89:0/498054483 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fafc80549d0 con 0x7fafd015bdd0 2023-11-24T06:20:26.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.045+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafd0067f60 con 0x7fafd015bdd0 2023-11-24T06:20:26.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.046+0000 7fafc6ffd700 1 -- 172.21.15.89:0/498054483 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fafc8057ff0 con 0x7fafd015bdd0 2023-11-24T06:20:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:26.048 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2011089843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2011089843},{"type":"v1","addr":"172.21.15.89:6836","nonce":2011089843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4955,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/2182962276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2182962276},{"type":"v1","addr":"172.21.15.123:6834","nonce":2182962276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2982219407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2982219407},{"type":"v1","addr":"172.21.15.123:6835","nonce":2982219407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:22.026873+0000","modified":"2023-11-24T06:20:25.053779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5577":{"gid":5577,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1341821445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1341821445},{"type":"v1","addr":"172.21.15.89:6837","nonce":1341821445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T06:20:26.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fafb006aed0 msgr2=0x7fafb006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fafb006aed0 0x7fafb006d390 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fafb8006fd0 tx=0x7fafb8008040 comp rx=0 tx=0).stop 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd015bdd0 msgr2=0x7fafd01e6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd015bdd0 0x7fafd01e6860 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fafc800cc10 tx=0x7fafc800cf20 comp rx=0 tx=0).stop 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 shutdown_connections 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd015bdd0 0x7fafd01e6860 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafd00b3370 0x7fafd0156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fafb006aed0 0x7fafb006d390 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 --2- 172.21.15.89:0/498054483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd00d1c70 0x7fafd00cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 >> 172.21.15.89:0/498054483 conn(0x7fafd00731a0 msgr2=0x7fafd00b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 shutdown_connections 2023-11-24T06:20:26.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.048+0000 7fafd4920700 1 -- 172.21.15.89:0/498054483 wait complete. 2023-11-24T06:20:26.052 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 31 2023-11-24T06:20:26.063 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:26.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.181+0000 7fdc37860700 1 Processor -- start 2023-11-24T06:20:26.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.181+0000 7fdc37860700 1 -- start start 2023-11-24T06:20:26.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.182+0000 7fdc37860700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdc3015dfc0 0x7fdc300d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.182+0000 7fdc37860700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdc300b2fc0 0x7fdc300d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.182+0000 7fdc37860700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc300d2d10 con 0x7fdc300b2a70 2023-11-24T06:20:26.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.182+0000 7fdc37860700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc300cefd0 con 0x7fdc3015dfc0 2023-11-24T06:20:26.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.182+0000 7fdc37860700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc300cf2b0 con 0x7fdc300b2fc0 2023-11-24T06:20:26.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.182+0000 7fdc355fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdc3015dfc0 0x7fdc300d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.182+0000 7fdc355fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdc3015dfc0 0x7fdc300d0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37946/0 (socket says 172.21.15.89:37946) 2023-11-24T06:20:26.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.182+0000 7fdc355fc700 1 -- 172.21.15.89:0/1190437018 learned_addr learned my addr 172.21.15.89:0/1190437018 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:26.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc355fc700 1 -- 172.21.15.89:0/1190437018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdc300b2fc0 msgr2=0x7fdc300d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:26.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc355fc700 1 --2- 172.21.15.89:0/1190437018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdc300b2fc0 0x7fdc300d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc355fc700 1 -- 172.21.15.89:0/1190437018 >> v1:172.21.15.123:6789/0 conn(0x7fdc300b2a70 legacy=0x7fdc30161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:26.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc355fc700 1 -- 172.21.15.89:0/1190437018 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc300d2fb0 con 0x7fdc3015dfc0 2023-11-24T06:20:26.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc355fc700 1 --2- 172.21.15.89:0/1190437018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3015dfc0 0x7fdc300d0e60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdc200098f0 tx=0x7fdc20009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caf2c75ee64a1c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc27fff700 1 -- 172.21.15.89:0/1190437018 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc20004230 con 0x7fdc3015dfc0 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc27fff700 1 -- 172.21.15.89:0/1190437018 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc20004390 con 0x7fdc3015dfc0 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc27fff700 1 -- 172.21.15.89:0/1190437018 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc200036a0 con 0x7fdc3015dfc0 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc37860700 1 -- 172.21.15.89:0/1190437018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3015dfc0 msgr2=0x7fdc300d0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc37860700 1 --2- 172.21.15.89:0/1190437018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3015dfc0 0x7fdc300d0e60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdc200098f0 tx=0x7fdc20009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.183+0000 7fdc37860700 1 -- 172.21.15.89:0/1190437018 shutdown_connections 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.184+0000 7fdc37860700 1 --2- 172.21.15.89:0/1190437018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3015dfc0 0x7fdc300d0e60 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.184+0000 7fdc37860700 1 --2- 172.21.15.89:0/1190437018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdc300b2fc0 0x7fdc300d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.184+0000 7fdc37860700 1 -- 172.21.15.89:0/1190437018 >> 172.21.15.89:0/1190437018 conn(0x7fdc3006b410 msgr2=0x7fdc3006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.184+0000 7fdc37860700 1 -- 172.21.15.89:0/1190437018 shutdown_connections 2023-11-24T06:20:26.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.184+0000 7fdc37860700 1 -- 172.21.15.89:0/1190437018 wait complete. 2023-11-24T06:20:26.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.185+0000 7fdc37860700 1 Processor -- start 2023-11-24T06:20:26.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.185+0000 7fdc37860700 1 -- start start 2023-11-24T06:20:26.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.185+0000 7fdc37860700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc300b2a70 0x7fdc30144fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.185+0000 7fdc37860700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdc300b2fc0 0x7fdc3015b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.185+0000 7fdc37860700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc3015dfc0 0x7fdc3015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc37860700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc30145500 con 0x7fdc300b2a70 2023-11-24T06:20:26.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc37860700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc300c95f0 con 0x7fdc3015dfc0 2023-11-24T06:20:26.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc37860700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc301458b0 con 0x7fdc300b2fc0 2023-11-24T06:20:26.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc300b2a70 0x7fdc30144fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc300b2a70 0x7fdc30144fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37952/0 (socket says 172.21.15.89:37952) 2023-11-24T06:20:26.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 -- 172.21.15.89:0/2158548856 learned_addr learned my addr 172.21.15.89:0/2158548856 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 -- 172.21.15.89:0/2158548856 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdc300b2fc0 msgr2=0x7fdc3015b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdc300b2fc0 0x7fdc3015b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 -- 172.21.15.89:0/2158548856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc3015dfc0 msgr2=0x7fdc3015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc3015dfc0 0x7fdc3015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 -- 172.21.15.89:0/2158548856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20009550 con 0x7fdc300b2a70 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc355fc700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc300b2a70 0x7fdc30144fc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdc300b5c40 tx=0x7fdc20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc25ffb700 1 -- 172.21.15.89:0/2158548856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc20003e70 con 0x7fdc300b2a70 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc25ffb700 1 -- 172.21.15.89:0/2158548856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2001f070 con 0x7fdc300b2a70 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc25ffb700 1 -- 172.21.15.89:0/2158548856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2000b8c0 con 0x7fdc300b2a70 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc30145b90 con 0x7fdc300b2a70 2023-11-24T06:20:26.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.186+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc30146150 con 0x7fdc300b2a70 2023-11-24T06:20:26.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.187+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc30074d60 con 0x7fdc300b2a70 2023-11-24T06:20:26.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.188+0000 7fdc25ffb700 1 -- 172.21.15.89:0/2158548856 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdc20021030 con 0x7fdc300b2a70 2023-11-24T06:20:26.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.188+0000 7fdc25ffb700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdc1406af00 0x7fdc1406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.188+0000 7fdc25ffb700 1 -- 172.21.15.89:0/2158548856 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc2008ac30 con 0x7fdc300b2a70 2023-11-24T06:20:26.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.188+0000 7fdc34dfb700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdc1406af00 0x7fdc1406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.189+0000 7fdc34dfb700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdc1406af00 0x7fdc1406d3c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc300cac30 tx=0x7fdc2c006d20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.192+0000 7fdc25ffb700 1 -- 172.21.15.89:0/2158548856 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc20055e90 con 0x7fdc300b2a70 2023-11-24T06:20:26.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.385+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc30074d60 con 0x7fdc300b2a70 2023-11-24T06:20:26.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.386+0000 7fdc25ffb700 1 -- 172.21.15.89:0/2158548856 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16529 (secure 0 0 0) 0x7fdc200594b0 con 0x7fdc300b2a70 2023-11-24T06:20:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:26.388 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":53,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:23.644194+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:20.413276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:20.978712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdc1406af00 msgr2=0x7fdc1406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdc1406af00 0x7fdc1406d3c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc300cac30 tx=0x7fdc2c006d20 comp rx=0 tx=0).stop 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc300b2a70 msgr2=0x7fdc30144fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc300b2a70 0x7fdc30144fc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdc300b5c40 tx=0x7fdc20004750 comp rx=0 tx=0).stop 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 shutdown_connections 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdc1406af00 0x7fdc1406d3c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc3015dfc0 0x7fdc3015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc300b2a70 0x7fdc30144fc0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 --2- 172.21.15.89:0/2158548856 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdc300b2fc0 0x7fdc3015b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.388+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 >> 172.21.15.89:0/2158548856 conn(0x7fdc3006b410 msgr2=0x7fdc30160530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.389+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 shutdown_connections 2023-11-24T06:20:26.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.389+0000 7fdc37860700 1 -- 172.21.15.89:0/2158548856 wait complete. 2023-11-24T06:20:26.403 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T06:20:26.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:20:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.522+0000 7f3e3f1d9700 1 Processor -- start 2023-11-24T06:20:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.522+0000 7f3e3f1d9700 1 -- start start 2023-11-24T06:20:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e3f1d9700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e38072df0 0x7f3e380af070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e3f1d9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3e380622a0 0x7f3e380af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e3f1d9700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380adfd0 con 0x7f3e3815c140 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e3f1d9700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380ae2b0 con 0x7f3e38072df0 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e3f1d9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380ae590 con 0x7f3e380622a0 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e37fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e38072df0 0x7f3e380af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e37fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e38072df0 0x7f3e380af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37962/0 (socket says 172.21.15.89:37962) 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e3cf75700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3e3815c140 0x7f3e380726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41464/0 (socket says 172.21.15.89:41464) 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e37fff700 1 -- 172.21.15.89:0/886608807 learned_addr learned my addr 172.21.15.89:0/886608807 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e3d776700 1 --2- 172.21.15.89:0/886608807 >> v2:172.21.15.123:3300/0 conn(0x7f3e380622a0 0x7f3e380af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e37fff700 1 -- 172.21.15.89:0/886608807 >> v2:172.21.15.123:3300/0 conn(0x7f3e380622a0 msgr2=0x7f3e380af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e37fff700 1 --2- 172.21.15.89:0/886608807 >> v2:172.21.15.123:3300/0 conn(0x7f3e380622a0 0x7f3e380af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e37fff700 1 -- 172.21.15.89:0/886608807 >> v1:172.21.15.89:6789/0 conn(0x7f3e3815c140 legacy=0x7f3e380726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e37fff700 1 -- 172.21.15.89:0/886608807 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380ae870 con 0x7f3e38072df0 2023-11-24T06:20:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e37fff700 1 --2- 172.21.15.89:0/886608807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38072df0 0x7f3e380af070 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3e380725d0 tx=0x7f3e24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e427b4ea4512078 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e377fe700 1 -- 172.21.15.89:0/886608807 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2400b870 con 0x7f3e38072df0 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.523+0000 7f3e377fe700 1 -- 172.21.15.89:0/886608807 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2400b9d0 con 0x7f3e38072df0 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e377fe700 1 -- 172.21.15.89:0/886608807 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2400bd60 con 0x7f3e38072df0 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/886608807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38072df0 msgr2=0x7f3e380af070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/886608807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38072df0 0x7f3e380af070 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3e380725d0 tx=0x7f3e24009bd0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/886608807 shutdown_connections 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/886608807 >> v2:172.21.15.123:3300/0 conn(0x7f3e380622a0 0x7f3e380af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/886608807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e38072df0 0x7f3e380af070 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/886608807 >> 172.21.15.89:0/886608807 conn(0x7f3e380b57b0 msgr2=0x7f3e380b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/886608807 shutdown_connections 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.524+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/886608807 wait complete. 2023-11-24T06:20:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.525+0000 7f3e3f1d9700 1 Processor -- start 2023-11-24T06:20:26.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.525+0000 7f3e3f1d9700 1 -- start start 2023-11-24T06:20:26.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3f1d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e380622a0 0x7f3e380b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3f1d9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e38072df0 0x7f3e380b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3f1d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3815c140 0x7f3e38155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3f1d9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380b4450 con 0x7f3e3815c140 2023-11-24T06:20:26.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3f1d9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380b3a80 con 0x7f3e380622a0 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3f1d9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38156080 con 0x7f3e38072df0 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3d776700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3815c140 0x7f3e38155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3d776700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3815c140 0x7f3e38155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37970/0 (socket says 172.21.15.89:37970) 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3d776700 1 -- 172.21.15.89:0/3785967263 learned_addr learned my addr 172.21.15.89:0/3785967263 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e3cf75700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e380622a0 0x7f3e380b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.526+0000 7f3e37fff700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e38072df0 0x7f3e380b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3d776700 1 -- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e38072df0 msgr2=0x7f3e380b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3d776700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e38072df0 0x7f3e380b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3d776700 1 -- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e380622a0 msgr2=0x7f3e380b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3d776700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e380622a0 0x7f3e380b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3d776700 1 -- 172.21.15.89:0/3785967263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e24009550 con 0x7f3e3815c140 2023-11-24T06:20:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3cf75700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e380622a0 0x7f3e380b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e37fff700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e38072df0 0x7f3e380b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3d776700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3815c140 0x7f3e38155b40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3e30007e40 tx=0x7f3e3000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e357fa700 1 -- 172.21.15.89:0/3785967263 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3000eea0 con 0x7f3e3815c140 2023-11-24T06:20:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e357fa700 1 -- 172.21.15.89:0/3785967263 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e30014420 con 0x7f3e3815c140 2023-11-24T06:20:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e38156300 con 0x7f3e3815c140 2023-11-24T06:20:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e357fa700 1 -- 172.21.15.89:0/3785967263 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e300147b0 con 0x7f3e3815c140 2023-11-24T06:20:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.527+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e381565e0 con 0x7f3e3815c140 2023-11-24T06:20:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.528+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e380b3c10 con 0x7f3e3815c140 2023-11-24T06:20:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.528+0000 7f3e357fa700 1 -- 172.21.15.89:0/3785967263 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3e30014910 con 0x7f3e3815c140 2023-11-24T06:20:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.529+0000 7f3e357fa700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.529+0000 7f3e3cf75700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.529+0000 7f3e357fa700 1 -- 172.21.15.89:0/3785967263 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e30089da0 con 0x7f3e3815c140 2023-11-24T06:20:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.529+0000 7f3e3cf75700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c000f60 tx=0x7f3e2c007dd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.533+0000 7f3e357fa700 1 -- 172.21.15.89:0/3785967263 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e30054e50 con 0x7f3e3815c140 2023-11-24T06:20:26.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.723+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e380adc80 con 0x7f3e1c06aed0 2023-11-24T06:20:26.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.731+0000 7f3e357fa700 1 -- 172.21.15.89:0/3785967263 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284268 (secure 0 0 0) 0x7f3e2c005cd0 con 0x7f3e1c06aed0 2023-11-24T06:20:26.736 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:26.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 msgr2=0x7f3e1c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c000f60 tx=0x7f3e2c007dd0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3815c140 msgr2=0x7f3e38155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3815c140 0x7f3e38155b40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3e30007e40 tx=0x7f3e3000e3c0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 shutdown_connections 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e3815c140 0x7f3e38155b40 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3e1c06aed0 0x7f3e1c06d390 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e380622a0 0x7f3e380b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 --2- 172.21.15.89:0/3785967263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e38072df0 0x7f3e380b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 >> 172.21.15.89:0/3785967263 conn(0x7f3e380b57b0 msgr2=0x7f3e3806e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 shutdown_connections 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.735+0000 7f3e3f1d9700 1 -- 172.21.15.89:0/3785967263 wait complete. 2023-11-24T06:20:26.739 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:20:26.749 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2023-11-24T06:20:25.738893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222024,"kb_used_data":5844,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752696,"statfs":{"total":773094113280,"available":772866760704,"internally_reserved":0,"allocated":5984256,"data_stored":3195611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.123598"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623555+0000","last_change":"2023-11-24T06:20:21.623555+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623555+0000","last_undegraded":"2023-11-24T06:20:21.623555+0000","last_fullsized":"2023-11-24T06:20:21.623555+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.628738+0000","last_change":"2023-11-24T06:20:21.628738+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.628738+0000","last_undegraded":"2023-11-24T06:20:21.628738+0000","last_fullsized":"2023-11-24T06:20:21.628738+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623537+0000","last_change":"2023-11-24T06:20:21.623537+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623537+0000","last_undegraded":"2023-11-24T06:20:21.623537+0000","last_fullsized":"2023-11-24T06:20:21.623537+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626348+0000","last_change":"2023-11-24T06:20:21.626348+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626348+0000","last_undegraded":"2023-11-24T06:20:21.626348+0000","last_fullsized":"2023-11-24T06:20:21.626348+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624458+0000","last_change":"2023-11-24T06:20:21.624458+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624458+0000","last_undegraded":"2023-11-24T06:20:21.624458+0000","last_fullsized":"2023-11-24T06:20:21.624458+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627324+0000","last_change":"2023-11-24T06:20:21.627324+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627324+0000","last_undegraded":"2023-11-24T06:20:21.627324+0000","last_fullsized":"2023-11-24T06:20:21.627324+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622653+0000","last_change":"2023-11-24T06:20:21.622653+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622653+0000","last_undegraded":"2023-11-24T06:20:21.622653+0000","last_fullsized":"2023-11-24T06:20:21.622653+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.629361+0000","last_change":"2023-11-24T06:20:21.629361+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.629361+0000","last_undegraded":"2023-11-24T06:20:21.629361+0000","last_fullsized":"2023-11-24T06:20:21.629361+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.620894+0000","last_change":"2023-11-24T06:20:21.620894+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.620894+0000","last_undegraded":"2023-11-24T06:20:21.620894+0000","last_fullsized":"2023-11-24T06:20:21.620894+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626826+0000","last_change":"2023-11-24T06:20:21.626826+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626826+0000","last_undegraded":"2023-11-24T06:20:21.626826+0000","last_fullsized":"2023-11-24T06:20:21.626826+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.630536+0000","last_change":"2023-11-24T06:20:21.630536+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.630536+0000","last_undegraded":"2023-11-24T06:20:21.630536+0000","last_fullsized":"2023-11-24T06:20:21.630536+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625573+0000","last_change":"2023-11-24T06:20:21.625573+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625573+0000","last_undegraded":"2023-11-24T06:20:21.625573+0000","last_fullsized":"2023-11-24T06:20:21.625573+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626066+0000","last_change":"2023-11-24T06:20:21.626066+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626066+0000","last_undegraded":"2023-11-24T06:20:21.626066+0000","last_fullsized":"2023-11-24T06:20:21.626066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622517+0000","last_change":"2023-11-24T06:20:21.622517+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622517+0000","last_undegraded":"2023-11-24T06:20:21.622517+0000","last_fullsized":"2023-11-24T06:20:21.622517+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.620882+0000","last_change":"2023-11-24T06:20:21.620882+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.620882+0000","last_undegraded":"2023-11-24T06:20:21.620882+0000","last_fullsized":"2023-11-24T06:20:21.620882+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624061+0000","last_change":"2023-11-24T06:20:21.624061+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624061+0000","last_undegraded":"2023-11-24T06:20:21.624061+0000","last_fullsized":"2023-11-24T06:20:21.624061+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626805+0000","last_change":"2023-11-24T06:20:21.626805+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626805+0000","last_undegraded":"2023-11-24T06:20:21.626805+0000","last_fullsized":"2023-11-24T06:20:21.626805+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623770+0000","last_change":"2023-11-24T06:20:21.623770+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623770+0000","last_undegraded":"2023-11-24T06:20:21.623770+0000","last_fullsized":"2023-11-24T06:20:21.623770+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624008+0000","last_change":"2023-11-24T06:20:21.624008+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624008+0000","last_undegraded":"2023-11-24T06:20:21.624008+0000","last_fullsized":"2023-11-24T06:20:21.624008+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625942+0000","last_change":"2023-11-24T06:20:21.625942+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625942+0000","last_undegraded":"2023-11-24T06:20:21.625942+0000","last_fullsized":"2023-11-24T06:20:21.625942+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623073+0000","last_change":"2023-11-24T06:20:21.623073+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623073+0000","last_undegraded":"2023-11-24T06:20:21.623073+0000","last_fullsized":"2023-11-24T06:20:21.623073+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625109+0000","last_change":"2023-11-24T06:20:21.625109+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625109+0000","last_undegraded":"2023-11-24T06:20:21.625109+0000","last_fullsized":"2023-11-24T06:20:21.625109+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622512+0000","last_change":"2023-11-24T06:20:21.622512+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622512+0000","last_undegraded":"2023-11-24T06:20:21.622512+0000","last_fullsized":"2023-11-24T06:20:21.622512+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627183+0000","last_change":"2023-11-24T06:20:21.627183+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627183+0000","last_undegraded":"2023-11-24T06:20:21.627183+0000","last_fullsized":"2023-11-24T06:20:21.627183+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623106+0000","last_change":"2023-11-24T06:20:21.623106+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623106+0000","last_undegraded":"2023-11-24T06:20:21.623106+0000","last_fullsized":"2023-11-24T06:20:21.623106+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625894+0000","last_change":"2023-11-24T06:20:21.625894+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625894+0000","last_undegraded":"2023-11-24T06:20:21.625894+0000","last_fullsized":"2023-11-24T06:20:21.625894+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627857+0000","last_change":"2023-11-24T06:20:21.627857+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627857+0000","last_undegraded":"2023-11-24T06:20:21.627857+0000","last_fullsized":"2023-11-24T06:20:21.627857+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622254+0000","last_change":"2023-11-24T06:20:21.622254+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622254+0000","last_undegraded":"2023-11-24T06:20:21.622254+0000","last_fullsized":"2023-11-24T06:20:21.622254+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625015+0000","last_change":"2023-11-24T06:20:21.625015+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625015+0000","last_undegraded":"2023-11-24T06:20:21.625015+0000","last_fullsized":"2023-11-24T06:20:21.625015+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621840+0000","last_change":"2023-11-24T06:20:21.621840+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621840+0000","last_undegraded":"2023-11-24T06:20:21.621840+0000","last_fullsized":"2023-11-24T06:20:21.621840+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621760+0000","last_change":"2023-11-24T06:20:21.621760+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621760+0000","last_undegraded":"2023-11-24T06:20:21.621760+0000","last_fullsized":"2023-11-24T06:20:21.621760+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624542+0000","last_change":"2023-11-24T06:20:21.624542+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624542+0000","last_undegraded":"2023-11-24T06:20:21.624542+0000","last_fullsized":"2023-11-24T06:20:21.624542+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626103+0000","last_change":"2023-11-24T06:20:21.626103+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626103+0000","last_undegraded":"2023-11-24T06:20:21.626103+0000","last_fullsized":"2023-11-24T06:20:21.626103+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640218+0000","last_change":"2023-11-24T06:20:21.635404+0000","last_active":"2023-11-24T06:20:21.640218+0000","last_peered":"2023-11-24T06:20:21.640218+0000","last_clean":"2023-11-24T06:20:21.640218+0000","last_became_active":"2023-11-24T06:20:21.635136+0000","last_became_peered":"2023-11-24T06:20:21.635136+0000","last_unstale":"2023-11-24T06:20:21.640218+0000","last_undegraded":"2023-11-24T06:20:21.640218+0000","last_fullsized":"2023-11-24T06:20:21.640218+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:32.580874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.631414+0000","last_change":"2023-11-24T06:20:21.631414+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.631414+0000","last_undegraded":"2023-11-24T06:20:21.631414+0000","last_fullsized":"2023-11-24T06:20:21.631414+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639627+0000","last_change":"2023-11-24T06:20:21.634801+0000","last_active":"2023-11-24T06:20:21.639627+0000","last_peered":"2023-11-24T06:20:21.639627+0000","last_clean":"2023-11-24T06:20:21.639627+0000","last_became_active":"2023-11-24T06:20:21.634620+0000","last_became_peered":"2023-11-24T06:20:21.634620+0000","last_unstale":"2023-11-24T06:20:21.639627+0000","last_undegraded":"2023-11-24T06:20:21.639627+0000","last_fullsized":"2023-11-24T06:20:21.639627+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:26.240404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625721+0000","last_change":"2023-11-24T06:20:21.625721+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625721+0000","last_undegraded":"2023-11-24T06:20:21.625721+0000","last_fullsized":"2023-11-24T06:20:21.625721+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.647968+0000","last_change":"2023-11-24T06:20:21.641256+0000","last_active":"2023-11-24T06:20:21.647968+0000","last_peered":"2023-11-24T06:20:21.647968+0000","last_clean":"2023-11-24T06:20:21.647968+0000","last_became_active":"2023-11-24T06:20:21.640930+0000","last_became_peered":"2023-11-24T06:20:21.640930+0000","last_unstale":"2023-11-24T06:20:21.647968+0000","last_undegraded":"2023-11-24T06:20:21.647968+0000","last_fullsized":"2023-11-24T06:20:21.647968+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:38.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627331+0000","last_change":"2023-11-24T06:20:21.627331+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627331+0000","last_undegraded":"2023-11-24T06:20:21.627331+0000","last_fullsized":"2023-11-24T06:20:21.627331+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638358+0000","last_change":"2023-11-24T06:20:21.634844+0000","last_active":"2023-11-24T06:20:21.638358+0000","last_peered":"2023-11-24T06:20:21.638358+0000","last_clean":"2023-11-24T06:20:21.638358+0000","last_became_active":"2023-11-24T06:20:21.634564+0000","last_became_peered":"2023-11-24T06:20:21.634564+0000","last_unstale":"2023-11-24T06:20:21.638358+0000","last_undegraded":"2023-11-24T06:20:21.638358+0000","last_fullsized":"2023-11-24T06:20:21.638358+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623259+0000","last_change":"2023-11-24T06:20:21.623259+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623259+0000","last_undegraded":"2023-11-24T06:20:21.623259+0000","last_fullsized":"2023-11-24T06:20:21.623259+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.633149+0000","last_change":"2023-11-24T06:20:21.629599+0000","last_active":"2023-11-24T06:20:21.633149+0000","last_peered":"2023-11-24T06:20:21.633149+0000","last_clean":"2023-11-24T06:20:21.633149+0000","last_became_active":"2023-11-24T06:20:21.629219+0000","last_became_peered":"2023-11-24T06:20:21.629219+0000","last_unstale":"2023-11-24T06:20:21.633149+0000","last_undegraded":"2023-11-24T06:20:21.633149+0000","last_fullsized":"2023-11-24T06:20:21.633149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:18.021884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623607+0000","last_change":"2023-11-24T06:20:21.623607+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623607+0000","last_undegraded":"2023-11-24T06:20:21.623607+0000","last_fullsized":"2023-11-24T06:20:21.623607+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640257+0000","last_change":"2023-11-24T06:20:21.635521+0000","last_active":"2023-11-24T06:20:21.640257+0000","last_peered":"2023-11-24T06:20:21.640257+0000","last_clean":"2023-11-24T06:20:21.640257+0000","last_became_active":"2023-11-24T06:20:21.635214+0000","last_became_peered":"2023-11-24T06:20:21.635214+0000","last_unstale":"2023-11-24T06:20:21.640257+0000","last_undegraded":"2023-11-24T06:20:21.640257+0000","last_fullsized":"2023-11-24T06:20:21.640257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:23.435615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625448+0000","last_change":"2023-11-24T06:20:21.625448+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625448+0000","last_undegraded":"2023-11-24T06:20:21.625448+0000","last_fullsized":"2023-11-24T06:20:21.625448+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638387+0000","last_change":"2023-11-24T06:20:21.634192+0000","last_active":"2023-11-24T06:20:21.638387+0000","last_peered":"2023-11-24T06:20:21.638387+0000","last_clean":"2023-11-24T06:20:21.638387+0000","last_became_active":"2023-11-24T06:20:21.633818+0000","last_became_peered":"2023-11-24T06:20:21.633818+0000","last_unstale":"2023-11-24T06:20:21.638387+0000","last_undegraded":"2023-11-24T06:20:21.638387+0000","last_fullsized":"2023-11-24T06:20:21.638387+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:02:30.378149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624564+0000","last_change":"2023-11-24T06:20:21.624564+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624564+0000","last_undegraded":"2023-11-24T06:20:21.624564+0000","last_fullsized":"2023-11-24T06:20:21.624564+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640823+0000","last_change":"2023-11-24T06:20:21.637491+0000","last_active":"2023-11-24T06:20:21.640823+0000","last_peered":"2023-11-24T06:20:21.640823+0000","last_clean":"2023-11-24T06:20:21.640823+0000","last_became_active":"2023-11-24T06:20:21.637173+0000","last_became_peered":"2023-11-24T06:20:21.637173+0000","last_unstale":"2023-11-24T06:20:21.640823+0000","last_undegraded":"2023-11-24T06:20:21.640823+0000","last_fullsized":"2023-11-24T06:20:21.640823+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:15:39.025211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621408+0000","last_change":"2023-11-24T06:20:21.621408+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621408+0000","last_undegraded":"2023-11-24T06:20:21.621408+0000","last_fullsized":"2023-11-24T06:20:21.621408+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640898+0000","last_change":"2023-11-24T06:20:21.638447+0000","last_active":"2023-11-24T06:20:21.640898+0000","last_peered":"2023-11-24T06:20:21.640898+0000","last_clean":"2023-11-24T06:20:21.640898+0000","last_became_active":"2023-11-24T06:20:21.638158+0000","last_became_peered":"2023-11-24T06:20:21.638158+0000","last_unstale":"2023-11-24T06:20:21.640898+0000","last_undegraded":"2023-11-24T06:20:21.640898+0000","last_fullsized":"2023-11-24T06:20:21.640898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:51.335391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621969+0000","last_change":"2023-11-24T06:20:21.621969+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621969+0000","last_undegraded":"2023-11-24T06:20:21.621969+0000","last_fullsized":"2023-11-24T06:20:21.621969+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638472+0000","last_change":"2023-11-24T06:20:21.631051+0000","last_active":"2023-11-24T06:20:21.638472+0000","last_peered":"2023-11-24T06:20:21.638472+0000","last_clean":"2023-11-24T06:20:21.638472+0000","last_became_active":"2023-11-24T06:20:21.630773+0000","last_became_peered":"2023-11-24T06:20:21.630773+0000","last_unstale":"2023-11-24T06:20:21.638472+0000","last_undegraded":"2023-11-24T06:20:21.638472+0000","last_fullsized":"2023-11-24T06:20:21.638472+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:19.355871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627272+0000","last_change":"2023-11-24T06:20:21.627272+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627272+0000","last_undegraded":"2023-11-24T06:20:21.627272+0000","last_fullsized":"2023-11-24T06:20:21.627272+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640832+0000","last_change":"2023-11-24T06:20:21.637050+0000","last_active":"2023-11-24T06:20:21.640832+0000","last_peered":"2023-11-24T06:20:21.640832+0000","last_clean":"2023-11-24T06:20:21.640832+0000","last_became_active":"2023-11-24T06:20:21.636769+0000","last_became_peered":"2023-11-24T06:20:21.636769+0000","last_unstale":"2023-11-24T06:20:21.640832+0000","last_undegraded":"2023-11-24T06:20:21.640832+0000","last_fullsized":"2023-11-24T06:20:21.640832+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:41:45.468254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621600+0000","last_change":"2023-11-24T06:20:21.621600+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621600+0000","last_undegraded":"2023-11-24T06:20:21.621600+0000","last_fullsized":"2023-11-24T06:20:21.621600+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640858+0000","last_change":"2023-11-24T06:20:21.638561+0000","last_active":"2023-11-24T06:20:21.640858+0000","last_peered":"2023-11-24T06:20:21.640858+0000","last_clean":"2023-11-24T06:20:21.640858+0000","last_became_active":"2023-11-24T06:20:21.638252+0000","last_became_peered":"2023-11-24T06:20:21.638252+0000","last_unstale":"2023-11-24T06:20:21.640858+0000","last_undegraded":"2023-11-24T06:20:21.640858+0000","last_fullsized":"2023-11-24T06:20:21.640858+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:25.850351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627692+0000","last_change":"2023-11-24T06:20:21.627692+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627692+0000","last_undegraded":"2023-11-24T06:20:21.627692+0000","last_fullsized":"2023-11-24T06:20:21.627692+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639371+0000","last_change":"2023-11-24T06:20:21.633330+0000","last_active":"2023-11-24T06:20:21.639371+0000","last_peered":"2023-11-24T06:20:21.639371+0000","last_clean":"2023-11-24T06:20:21.639371+0000","last_became_active":"2023-11-24T06:20:21.633057+0000","last_became_peered":"2023-11-24T06:20:21.633057+0000","last_unstale":"2023-11-24T06:20:21.639371+0000","last_undegraded":"2023-11-24T06:20:21.639371+0000","last_fullsized":"2023-11-24T06:20:21.639371+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:29.077158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622661+0000","last_change":"2023-11-24T06:20:21.622661+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622661+0000","last_undegraded":"2023-11-24T06:20:21.622661+0000","last_fullsized":"2023-11-24T06:20:21.622661+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.633740+0000","last_change":"2023-11-24T06:20:21.632526+0000","last_active":"2023-11-24T06:20:21.633740+0000","last_peered":"2023-11-24T06:20:21.633740+0000","last_clean":"2023-11-24T06:20:21.633740+0000","last_became_active":"2023-11-24T06:20:21.632218+0000","last_became_peered":"2023-11-24T06:20:21.632218+0000","last_unstale":"2023-11-24T06:20:21.633740+0000","last_undegraded":"2023-11-24T06:20:21.633740+0000","last_fullsized":"2023-11-24T06:20:21.633740+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:57:23.439486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621256+0000","last_change":"2023-11-24T06:20:21.621256+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621256+0000","last_undegraded":"2023-11-24T06:20:21.621256+0000","last_fullsized":"2023-11-24T06:20:21.621256+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638406+0000","last_change":"2023-11-24T06:20:21.635534+0000","last_active":"2023-11-24T06:20:21.638406+0000","last_peered":"2023-11-24T06:20:21.638406+0000","last_clean":"2023-11-24T06:20:21.638406+0000","last_became_active":"2023-11-24T06:20:21.635099+0000","last_became_peered":"2023-11-24T06:20:21.635099+0000","last_unstale":"2023-11-24T06:20:21.638406+0000","last_undegraded":"2023-11-24T06:20:21.638406+0000","last_fullsized":"2023-11-24T06:20:21.638406+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:18:34.137206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621164+0000","last_change":"2023-11-24T06:20:21.621164+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621164+0000","last_undegraded":"2023-11-24T06:20:21.621164+0000","last_fullsized":"2023-11-24T06:20:21.621164+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.636688+0000","last_change":"2023-11-24T06:20:21.634951+0000","last_active":"2023-11-24T06:20:21.636688+0000","last_peered":"2023-11-24T06:20:21.636688+0000","last_clean":"2023-11-24T06:20:21.636688+0000","last_became_active":"2023-11-24T06:20:21.634653+0000","last_became_peered":"2023-11-24T06:20:21.634653+0000","last_unstale":"2023-11-24T06:20:21.636688+0000","last_undegraded":"2023-11-24T06:20:21.636688+0000","last_fullsized":"2023-11-24T06:20:21.636688+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:30.394419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621410+0000","last_change":"2023-11-24T06:20:21.621410+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621410+0000","last_undegraded":"2023-11-24T06:20:21.621410+0000","last_fullsized":"2023-11-24T06:20:21.621410+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639474+0000","last_change":"2023-11-24T06:20:21.633653+0000","last_active":"2023-11-24T06:20:21.639474+0000","last_peered":"2023-11-24T06:20:21.639474+0000","last_clean":"2023-11-24T06:20:21.639474+0000","last_became_active":"2023-11-24T06:20:21.633333+0000","last_became_peered":"2023-11-24T06:20:21.633333+0000","last_unstale":"2023-11-24T06:20:21.639474+0000","last_undegraded":"2023-11-24T06:20:21.639474+0000","last_fullsized":"2023-11-24T06:20:21.639474+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:44.134108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627180+0000","last_change":"2023-11-24T06:20:21.627180+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627180+0000","last_undegraded":"2023-11-24T06:20:21.627180+0000","last_fullsized":"2023-11-24T06:20:21.627180+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640570+0000","last_change":"2023-11-24T06:20:21.638281+0000","last_active":"2023-11-24T06:20:21.640570+0000","last_peered":"2023-11-24T06:20:21.640570+0000","last_clean":"2023-11-24T06:20:21.640570+0000","last_became_active":"2023-11-24T06:20:21.638014+0000","last_became_peered":"2023-11-24T06:20:21.638014+0000","last_unstale":"2023-11-24T06:20:21.640570+0000","last_undegraded":"2023-11-24T06:20:21.640570+0000","last_fullsized":"2023-11-24T06:20:21.640570+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:15:38.645013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623054+0000","last_change":"2023-11-24T06:20:21.623054+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623054+0000","last_undegraded":"2023-11-24T06:20:21.623054+0000","last_fullsized":"2023-11-24T06:20:21.623054+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638275+0000","last_change":"2023-11-24T06:20:21.634745+0000","last_active":"2023-11-24T06:20:21.638275+0000","last_peered":"2023-11-24T06:20:21.638275+0000","last_clean":"2023-11-24T06:20:21.638275+0000","last_became_active":"2023-11-24T06:20:21.634453+0000","last_became_peered":"2023-11-24T06:20:21.634453+0000","last_unstale":"2023-11-24T06:20:21.638275+0000","last_undegraded":"2023-11-24T06:20:21.638275+0000","last_fullsized":"2023-11-24T06:20:21.638275+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:24:57.040169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621052+0000","last_change":"2023-11-24T06:20:21.621052+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621052+0000","last_undegraded":"2023-11-24T06:20:21.621052+0000","last_fullsized":"2023-11-24T06:20:21.621052+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.642800+0000","last_change":"2023-11-24T06:20:21.640109+0000","last_active":"2023-11-24T06:20:21.642800+0000","last_peered":"2023-11-24T06:20:21.642800+0000","last_clean":"2023-11-24T06:20:21.642800+0000","last_became_active":"2023-11-24T06:20:21.639777+0000","last_became_peered":"2023-11-24T06:20:21.639777+0000","last_unstale":"2023-11-24T06:20:21.642800+0000","last_undegraded":"2023-11-24T06:20:21.642800+0000","last_fullsized":"2023-11-24T06:20:21.642800+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625137+0000","last_change":"2023-11-24T06:20:21.625137+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625137+0000","last_undegraded":"2023-11-24T06:20:21.625137+0000","last_fullsized":"2023-11-24T06:20:21.625137+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638300+0000","last_change":"2023-11-24T06:20:21.631857+0000","last_active":"2023-11-24T06:20:21.638300+0000","last_peered":"2023-11-24T06:20:21.638300+0000","last_clean":"2023-11-24T06:20:21.638300+0000","last_became_active":"2023-11-24T06:20:21.631396+0000","last_became_peered":"2023-11-24T06:20:21.631396+0000","last_unstale":"2023-11-24T06:20:21.638300+0000","last_undegraded":"2023-11-24T06:20:21.638300+0000","last_fullsized":"2023-11-24T06:20:21.638300+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:43.904787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625641+0000","last_change":"2023-11-24T06:20:21.625641+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625641+0000","last_undegraded":"2023-11-24T06:20:21.625641+0000","last_fullsized":"2023-11-24T06:20:21.625641+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.635183+0000","last_change":"2023-11-24T06:20:21.633654+0000","last_active":"2023-11-24T06:20:21.635183+0000","last_peered":"2023-11-24T06:20:21.635183+0000","last_clean":"2023-11-24T06:20:21.635183+0000","last_became_active":"2023-11-24T06:20:21.633387+0000","last_became_peered":"2023-11-24T06:20:21.633387+0000","last_unstale":"2023-11-24T06:20:21.635183+0000","last_undegraded":"2023-11-24T06:20:21.635183+0000","last_fullsized":"2023-11-24T06:20:21.635183+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:05.063980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625225+0000","last_change":"2023-11-24T06:20:21.625225+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625225+0000","last_undegraded":"2023-11-24T06:20:21.625225+0000","last_fullsized":"2023-11-24T06:20:21.625225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.641065+0000","last_change":"2023-11-24T06:20:21.639872+0000","last_active":"2023-11-24T06:20:21.641065+0000","last_peered":"2023-11-24T06:20:21.641065+0000","last_clean":"2023-11-24T06:20:21.641065+0000","last_became_active":"2023-11-24T06:20:21.639600+0000","last_became_peered":"2023-11-24T06:20:21.639600+0000","last_unstale":"2023-11-24T06:20:21.641065+0000","last_undegraded":"2023-11-24T06:20:21.641065+0000","last_fullsized":"2023-11-24T06:20:21.641065+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:15.519175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625436+0000","last_change":"2023-11-24T06:20:21.625436+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625436+0000","last_undegraded":"2023-11-24T06:20:21.625436+0000","last_fullsized":"2023-11-24T06:20:21.625436+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639363+0000","last_change":"2023-11-24T06:20:21.638221+0000","last_active":"2023-11-24T06:20:21.639363+0000","last_peered":"2023-11-24T06:20:21.639363+0000","last_clean":"2023-11-24T06:20:21.639363+0000","last_became_active":"2023-11-24T06:20:21.637800+0000","last_became_peered":"2023-11-24T06:20:21.637800+0000","last_unstale":"2023-11-24T06:20:21.639363+0000","last_undegraded":"2023-11-24T06:20:21.639363+0000","last_fullsized":"2023-11-24T06:20:21.639363+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:23.447147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.628285+0000","last_change":"2023-11-24T06:20:21.628285+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.628285+0000","last_undegraded":"2023-11-24T06:20:21.628285+0000","last_fullsized":"2023-11-24T06:20:21.628285+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640027+0000","last_change":"2023-11-24T06:20:21.635897+0000","last_active":"2023-11-24T06:20:21.640027+0000","last_peered":"2023-11-24T06:20:21.640027+0000","last_clean":"2023-11-24T06:20:21.640027+0000","last_became_active":"2023-11-24T06:20:21.635167+0000","last_became_peered":"2023-11-24T06:20:21.635167+0000","last_unstale":"2023-11-24T06:20:21.640027+0000","last_undegraded":"2023-11-24T06:20:21.640027+0000","last_fullsized":"2023-11-24T06:20:21.640027+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:54:05.115719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626755+0000","last_change":"2023-11-24T06:20:21.626755+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626755+0000","last_undegraded":"2023-11-24T06:20:21.626755+0000","last_fullsized":"2023-11-24T06:20:21.626755+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638367+0000","last_change":"2023-11-24T06:20:21.633952+0000","last_active":"2023-11-24T06:20:21.638367+0000","last_peered":"2023-11-24T06:20:21.638367+0000","last_clean":"2023-11-24T06:20:21.638367+0000","last_became_active":"2023-11-24T06:20:21.633629+0000","last_became_peered":"2023-11-24T06:20:21.633629+0000","last_unstale":"2023-11-24T06:20:21.638367+0000","last_undegraded":"2023-11-24T06:20:21.638367+0000","last_fullsized":"2023-11-24T06:20:21.638367+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:53.363443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626539+0000","last_change":"2023-11-24T06:20:21.626539+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626539+0000","last_undegraded":"2023-11-24T06:20:21.626539+0000","last_fullsized":"2023-11-24T06:20:21.626539+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638377+0000","last_change":"2023-11-24T06:20:21.634883+0000","last_active":"2023-11-24T06:20:21.638377+0000","last_peered":"2023-11-24T06:20:21.638377+0000","last_clean":"2023-11-24T06:20:21.638377+0000","last_became_active":"2023-11-24T06:20:21.634624+0000","last_became_peered":"2023-11-24T06:20:21.634624+0000","last_unstale":"2023-11-24T06:20:21.638377+0000","last_undegraded":"2023-11-24T06:20:21.638377+0000","last_fullsized":"2023-11-24T06:20:21.638377+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:38.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622724+0000","last_change":"2023-11-24T06:20:21.622724+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622724+0000","last_undegraded":"2023-11-24T06:20:21.622724+0000","last_fullsized":"2023-11-24T06:20:21.622724+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.637356+0000","last_change":"2023-11-24T06:20:21.635390+0000","last_active":"2023-11-24T06:20:21.637356+0000","last_peered":"2023-11-24T06:20:21.637356+0000","last_clean":"2023-11-24T06:20:21.637356+0000","last_became_active":"2023-11-24T06:20:21.635114+0000","last_became_peered":"2023-11-24T06:20:21.635114+0000","last_unstale":"2023-11-24T06:20:21.637356+0000","last_undegraded":"2023-11-24T06:20:21.637356+0000","last_fullsized":"2023-11-24T06:20:21.637356+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:15:39.025211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621069+0000","last_change":"2023-11-24T06:20:21.621069+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621069+0000","last_undegraded":"2023-11-24T06:20:21.621069+0000","last_fullsized":"2023-11-24T06:20:21.621069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640811+0000","last_change":"2023-11-24T06:20:21.636357+0000","last_active":"2023-11-24T06:20:21.640811+0000","last_peered":"2023-11-24T06:20:21.640811+0000","last_clean":"2023-11-24T06:20:21.640811+0000","last_became_active":"2023-11-24T06:20:21.636068+0000","last_became_peered":"2023-11-24T06:20:21.636068+0000","last_unstale":"2023-11-24T06:20:21.640811+0000","last_undegraded":"2023-11-24T06:20:21.640811+0000","last_fullsized":"2023-11-24T06:20:21.640811+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:24.432980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626644+0000","last_change":"2023-11-24T06:20:21.626644+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626644+0000","last_undegraded":"2023-11-24T06:20:21.626644+0000","last_fullsized":"2023-11-24T06:20:21.626644+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.641635+0000","last_change":"2023-11-24T06:20:21.641031+0000","last_active":"2023-11-24T06:20:21.641635+0000","last_peered":"2023-11-24T06:20:21.641635+0000","last_clean":"2023-11-24T06:20:21.641635+0000","last_became_active":"2023-11-24T06:20:21.640826+0000","last_became_peered":"2023-11-24T06:20:21.640826+0000","last_unstale":"2023-11-24T06:20:21.641635+0000","last_undegraded":"2023-11-24T06:20:21.641635+0000","last_fullsized":"2023-11-24T06:20:21.641635+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:20.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622735+0000","last_change":"2023-11-24T06:20:21.622735+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622735+0000","last_undegraded":"2023-11-24T06:20:21.622735+0000","last_fullsized":"2023-11-24T06:20:21.622735+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639612+0000","last_change":"2023-11-24T06:20:21.636646+0000","last_active":"2023-11-24T06:20:21.639612+0000","last_peered":"2023-11-24T06:20:21.639612+0000","last_clean":"2023-11-24T06:20:21.639612+0000","last_became_active":"2023-11-24T06:20:21.636335+0000","last_became_peered":"2023-11-24T06:20:21.636335+0000","last_unstale":"2023-11-24T06:20:21.639612+0000","last_undegraded":"2023-11-24T06:20:21.639612+0000","last_fullsized":"2023-11-24T06:20:21.639612+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623457+0000","last_change":"2023-11-24T06:20:21.623457+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623457+0000","last_undegraded":"2023-11-24T06:20:21.623457+0000","last_fullsized":"2023-11-24T06:20:21.623457+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638494+0000","last_change":"2023-11-24T06:20:21.635744+0000","last_active":"2023-11-24T06:20:21.638494+0000","last_peered":"2023-11-24T06:20:21.638494+0000","last_clean":"2023-11-24T06:20:21.638494+0000","last_became_active":"2023-11-24T06:20:21.635447+0000","last_became_peered":"2023-11-24T06:20:21.635447+0000","last_unstale":"2023-11-24T06:20:21.638494+0000","last_undegraded":"2023-11-24T06:20:21.638494+0000","last_fullsized":"2023-11-24T06:20:21.638494+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:17.983184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-11-24T06:20:20.618880+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:20.618880+0000","last_peered":"2023-11-24T06:20:20.618880+0000","last_clean":"2023-11-24T06:20:20.618880+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:20.618880+0000","last_undegraded":"2023-11-24T06:20:20.618880+0000","last_fullsized":"2023-11-24T06:20:20.618880+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":841444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.70899999999999996}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.81000000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.78000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":6.423,"5min":6.423,"15min":6.423},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":12.554,"5min":12.554,"15min":12.554},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.75700000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.81599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":257343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":841444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.255}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.30599999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.373}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.61199999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56200000000000006}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.57299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.41799999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.37},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.58799999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.89200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.223}]},{"osd":1,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.246},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.28899999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.317},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.30199999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.434},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53500000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.627},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67500000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.66200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.04,"5min":3.04,"15min":3.04},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":10.198,"5min":10.198,"15min":10.198},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":10.215999999999999,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.63500000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.66000000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:20:26.754 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:20:26.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.910+0000 7fd3e2f66700 1 Processor -- start 2023-11-24T06:20:26.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.910+0000 7fd3e2f66700 1 -- start start 2023-11-24T06:20:26.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e2f66700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc0adf00 0x7fd3dc06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e2f66700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd3dc06c540 0x7fd3dc0b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e2f66700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3dc0af950 con 0x7fd3dc1446a0 2023-11-24T06:20:26.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e2f66700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3dc144a80 con 0x7fd3dc06c540 2023-11-24T06:20:26.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e2f66700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc06c9b0 con 0x7fd3dc0adf00 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e1503700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd3dc06c540 0x7fd3dc0b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e1503700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd3dc06c540 0x7fd3dc0b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37978/0 (socket says 172.21.15.89:37978) 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e1503700 1 -- 172.21.15.89:0/2076967361 learned_addr learned my addr 172.21.15.89:0/2076967361 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e0d02700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd3dc1446a0 0x7fd3dc0b5ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41478/0 (socket says 172.21.15.89:41478) 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3dbfff700 1 --2- 172.21.15.89:0/2076967361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc0adf00 0x7fd3dc06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e1503700 1 -- 172.21.15.89:0/2076967361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc0adf00 msgr2=0x7fd3dc06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e1503700 1 --2- 172.21.15.89:0/2076967361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc0adf00 0x7fd3dc06c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e1503700 1 -- 172.21.15.89:0/2076967361 >> v1:172.21.15.89:6789/0 conn(0x7fd3dc1446a0 legacy=0x7fd3dc0b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e1503700 1 -- 172.21.15.89:0/2076967361 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc0afc30 con 0x7fd3dc06c540 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3db7fe700 1 -- 172.21.15.89:0/2076967361 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699524398 0 0) 0x7fd3dc0af950 con 0x7fd3dc1446a0 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.911+0000 7fd3e1503700 1 --2- 172.21.15.89:0/2076967361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc06c540 0x7fd3dc0b3d40 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009e80 tx=0x7fd3d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dab841db83c6b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3db7fe700 1 -- 172.21.15.89:0/2076967361 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d400e070 con 0x7fd3dc06c540 2023-11-24T06:20:26.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3db7fe700 1 -- 172.21.15.89:0/2076967361 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d400b980 con 0x7fd3dc06c540 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3db7fe700 1 -- 172.21.15.89:0/2076967361 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d400bd30 con 0x7fd3dc06c540 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3e2f66700 1 -- 172.21.15.89:0/2076967361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc06c540 msgr2=0x7fd3dc0b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/2076967361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc06c540 0x7fd3dc0b3d40 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009e80 tx=0x7fd3d400b330 comp rx=0 tx=0).stop 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3e2f66700 1 -- 172.21.15.89:0/2076967361 shutdown_connections 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/2076967361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc06c540 0x7fd3dc0b3d40 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/2076967361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc0adf00 0x7fd3dc06c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3e2f66700 1 -- 172.21.15.89:0/2076967361 >> 172.21.15.89:0/2076967361 conn(0x7fd3dc071b90 msgr2=0x7fd3dc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3e2f66700 1 -- 172.21.15.89:0/2076967361 shutdown_connections 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.912+0000 7fd3e2f66700 1 -- 172.21.15.89:0/2076967361 wait complete. 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.913+0000 7fd3e2f66700 1 Processor -- start 2023-11-24T06:20:26.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.913+0000 7fd3e2f66700 1 -- start start 2023-11-24T06:20:26.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e2f66700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc06c540 0x7fd3dc158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e2f66700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc0adf00 0x7fd3dc0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e2f66700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc1446a0 0x7fd3dc13e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e2f66700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc13c190 con 0x7fd3dc1446a0 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e2f66700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc1ef420 con 0x7fd3dc0adf00 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e2f66700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc138c60 con 0x7fd3dc06c540 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e1503700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc1446a0 0x7fd3dc13e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e1503700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc1446a0 0x7fd3dc13e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37984/0 (socket says 172.21.15.89:37984) 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.914+0000 7fd3e1503700 1 -- 172.21.15.89:0/3046202083 learned_addr learned my addr 172.21.15.89:0/3046202083 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3dbfff700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc0adf00 0x7fd3dc0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e0d02700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc06c540 0x7fd3dc158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e1503700 1 -- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc06c540 msgr2=0x7fd3dc158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e1503700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc06c540 0x7fd3dc158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e1503700 1 -- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc0adf00 msgr2=0x7fd3dc0b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e1503700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc0adf00 0x7fd3dc0b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e1503700 1 -- 172.21.15.89:0/3046202083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d4009b30 con 0x7fd3dc1446a0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3dbfff700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc0adf00 0x7fd3dc0b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e1503700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc1446a0 0x7fd3dc13e710 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4008e50 tx=0x7fd3d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3d97fa700 1 -- 172.21.15.89:0/3046202083 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d40099b0 con 0x7fd3dc1446a0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3d97fa700 1 -- 172.21.15.89:0/3046202083 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d400bd00 con 0x7fd3dc1446a0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc140700 con 0x7fd3dc1446a0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3d97fa700 1 -- 172.21.15.89:0/3046202083 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d401c640 con 0x7fd3dc1446a0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.915+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3dc140cc0 con 0x7fd3dc1446a0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.916+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3dc0753f0 con 0x7fd3dc1446a0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.916+0000 7fd3d97fa700 1 -- 172.21.15.89:0/3046202083 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd3d401c7a0 con 0x7fd3dc1446a0 2023-11-24T06:20:26.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.917+0000 7fd3d97fa700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd3c006aed0 0x7fd3c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:26.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.917+0000 7fd3d97fa700 1 -- 172.21.15.89:0/3046202083 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3d408ac40 con 0x7fd3dc1446a0 2023-11-24T06:20:26.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.917+0000 7fd3e0d02700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd3c006aed0 0x7fd3c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:26.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.917+0000 7fd3e0d02700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd3c006aed0 0x7fd3c006d390 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0000f60 tx=0x7fd3d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:26.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:26.921+0000 7fd3d97fa700 1 -- 172.21.15.89:0/3046202083 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3d40598d0 con 0x7fd3dc1446a0 2023-11-24T06:20:27.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.105+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3dc0b04d0 con 0x7fd3c006aed0 2023-11-24T06:20:27.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.113+0000 7fd3d97fa700 1 -- 172.21.15.89:0/3046202083 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284268 (secure 0 0 0) 0x7fd3d00070d0 con 0x7fd3c006aed0 2023-11-24T06:20:27.119 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:27.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd3c006aed0 msgr2=0x7fd3c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd3c006aed0 0x7fd3c006d390 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0000f60 tx=0x7fd3d0009040 comp rx=0 tx=0).stop 2023-11-24T06:20:27.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc1446a0 msgr2=0x7fd3dc13e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc1446a0 0x7fd3dc13e710 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4008e50 tx=0x7fd3d4004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 shutdown_connections 2023-11-24T06:20:27.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3dc1446a0 0x7fd3dc13e710 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3dc06c540 0x7fd3dc158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd3c006aed0 0x7fd3c006d390 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 --2- 172.21.15.89:0/3046202083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3dc0adf00 0x7fd3dc0b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 >> 172.21.15.89:0/3046202083 conn(0x7fd3dc071b90 msgr2=0x7fd3dc0b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:27.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 shutdown_connections 2023-11-24T06:20:27.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.116+0000 7fd3e2f66700 1 -- 172.21.15.89:0/3046202083 wait complete. 2023-11-24T06:20:27.123 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:20:27.131 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2023-11-24T06:20:25.738893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222024,"kb_used_data":5844,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752696,"statfs":{"total":773094113280,"available":772866760704,"internally_reserved":0,"allocated":5984256,"data_stored":3195611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.123598"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623555+0000","last_change":"2023-11-24T06:20:21.623555+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623555+0000","last_undegraded":"2023-11-24T06:20:21.623555+0000","last_fullsized":"2023-11-24T06:20:21.623555+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.628738+0000","last_change":"2023-11-24T06:20:21.628738+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.628738+0000","last_undegraded":"2023-11-24T06:20:21.628738+0000","last_fullsized":"2023-11-24T06:20:21.628738+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623537+0000","last_change":"2023-11-24T06:20:21.623537+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623537+0000","last_undegraded":"2023-11-24T06:20:21.623537+0000","last_fullsized":"2023-11-24T06:20:21.623537+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626348+0000","last_change":"2023-11-24T06:20:21.626348+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626348+0000","last_undegraded":"2023-11-24T06:20:21.626348+0000","last_fullsized":"2023-11-24T06:20:21.626348+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624458+0000","last_change":"2023-11-24T06:20:21.624458+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624458+0000","last_undegraded":"2023-11-24T06:20:21.624458+0000","last_fullsized":"2023-11-24T06:20:21.624458+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627324+0000","last_change":"2023-11-24T06:20:21.627324+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627324+0000","last_undegraded":"2023-11-24T06:20:21.627324+0000","last_fullsized":"2023-11-24T06:20:21.627324+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622653+0000","last_change":"2023-11-24T06:20:21.622653+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622653+0000","last_undegraded":"2023-11-24T06:20:21.622653+0000","last_fullsized":"2023-11-24T06:20:21.622653+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.629361+0000","last_change":"2023-11-24T06:20:21.629361+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.629361+0000","last_undegraded":"2023-11-24T06:20:21.629361+0000","last_fullsized":"2023-11-24T06:20:21.629361+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.620894+0000","last_change":"2023-11-24T06:20:21.620894+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.620894+0000","last_undegraded":"2023-11-24T06:20:21.620894+0000","last_fullsized":"2023-11-24T06:20:21.620894+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626826+0000","last_change":"2023-11-24T06:20:21.626826+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626826+0000","last_undegraded":"2023-11-24T06:20:21.626826+0000","last_fullsized":"2023-11-24T06:20:21.626826+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.630536+0000","last_change":"2023-11-24T06:20:21.630536+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.630536+0000","last_undegraded":"2023-11-24T06:20:21.630536+0000","last_fullsized":"2023-11-24T06:20:21.630536+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625573+0000","last_change":"2023-11-24T06:20:21.625573+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625573+0000","last_undegraded":"2023-11-24T06:20:21.625573+0000","last_fullsized":"2023-11-24T06:20:21.625573+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626066+0000","last_change":"2023-11-24T06:20:21.626066+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626066+0000","last_undegraded":"2023-11-24T06:20:21.626066+0000","last_fullsized":"2023-11-24T06:20:21.626066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622517+0000","last_change":"2023-11-24T06:20:21.622517+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622517+0000","last_undegraded":"2023-11-24T06:20:21.622517+0000","last_fullsized":"2023-11-24T06:20:21.622517+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.620882+0000","last_change":"2023-11-24T06:20:21.620882+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.620882+0000","last_undegraded":"2023-11-24T06:20:21.620882+0000","last_fullsized":"2023-11-24T06:20:21.620882+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624061+0000","last_change":"2023-11-24T06:20:21.624061+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624061+0000","last_undegraded":"2023-11-24T06:20:21.624061+0000","last_fullsized":"2023-11-24T06:20:21.624061+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626805+0000","last_change":"2023-11-24T06:20:21.626805+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626805+0000","last_undegraded":"2023-11-24T06:20:21.626805+0000","last_fullsized":"2023-11-24T06:20:21.626805+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623770+0000","last_change":"2023-11-24T06:20:21.623770+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623770+0000","last_undegraded":"2023-11-24T06:20:21.623770+0000","last_fullsized":"2023-11-24T06:20:21.623770+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624008+0000","last_change":"2023-11-24T06:20:21.624008+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624008+0000","last_undegraded":"2023-11-24T06:20:21.624008+0000","last_fullsized":"2023-11-24T06:20:21.624008+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625942+0000","last_change":"2023-11-24T06:20:21.625942+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625942+0000","last_undegraded":"2023-11-24T06:20:21.625942+0000","last_fullsized":"2023-11-24T06:20:21.625942+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623073+0000","last_change":"2023-11-24T06:20:21.623073+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623073+0000","last_undegraded":"2023-11-24T06:20:21.623073+0000","last_fullsized":"2023-11-24T06:20:21.623073+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625109+0000","last_change":"2023-11-24T06:20:21.625109+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625109+0000","last_undegraded":"2023-11-24T06:20:21.625109+0000","last_fullsized":"2023-11-24T06:20:21.625109+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622512+0000","last_change":"2023-11-24T06:20:21.622512+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622512+0000","last_undegraded":"2023-11-24T06:20:21.622512+0000","last_fullsized":"2023-11-24T06:20:21.622512+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627183+0000","last_change":"2023-11-24T06:20:21.627183+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627183+0000","last_undegraded":"2023-11-24T06:20:21.627183+0000","last_fullsized":"2023-11-24T06:20:21.627183+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623106+0000","last_change":"2023-11-24T06:20:21.623106+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623106+0000","last_undegraded":"2023-11-24T06:20:21.623106+0000","last_fullsized":"2023-11-24T06:20:21.623106+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625894+0000","last_change":"2023-11-24T06:20:21.625894+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625894+0000","last_undegraded":"2023-11-24T06:20:21.625894+0000","last_fullsized":"2023-11-24T06:20:21.625894+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627857+0000","last_change":"2023-11-24T06:20:21.627857+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627857+0000","last_undegraded":"2023-11-24T06:20:21.627857+0000","last_fullsized":"2023-11-24T06:20:21.627857+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622254+0000","last_change":"2023-11-24T06:20:21.622254+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622254+0000","last_undegraded":"2023-11-24T06:20:21.622254+0000","last_fullsized":"2023-11-24T06:20:21.622254+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625015+0000","last_change":"2023-11-24T06:20:21.625015+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625015+0000","last_undegraded":"2023-11-24T06:20:21.625015+0000","last_fullsized":"2023-11-24T06:20:21.625015+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621840+0000","last_change":"2023-11-24T06:20:21.621840+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621840+0000","last_undegraded":"2023-11-24T06:20:21.621840+0000","last_fullsized":"2023-11-24T06:20:21.621840+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621760+0000","last_change":"2023-11-24T06:20:21.621760+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621760+0000","last_undegraded":"2023-11-24T06:20:21.621760+0000","last_fullsized":"2023-11-24T06:20:21.621760+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624542+0000","last_change":"2023-11-24T06:20:21.624542+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624542+0000","last_undegraded":"2023-11-24T06:20:21.624542+0000","last_fullsized":"2023-11-24T06:20:21.624542+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626103+0000","last_change":"2023-11-24T06:20:21.626103+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626103+0000","last_undegraded":"2023-11-24T06:20:21.626103+0000","last_fullsized":"2023-11-24T06:20:21.626103+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640218+0000","last_change":"2023-11-24T06:20:21.635404+0000","last_active":"2023-11-24T06:20:21.640218+0000","last_peered":"2023-11-24T06:20:21.640218+0000","last_clean":"2023-11-24T06:20:21.640218+0000","last_became_active":"2023-11-24T06:20:21.635136+0000","last_became_peered":"2023-11-24T06:20:21.635136+0000","last_unstale":"2023-11-24T06:20:21.640218+0000","last_undegraded":"2023-11-24T06:20:21.640218+0000","last_fullsized":"2023-11-24T06:20:21.640218+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:32.580874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.631414+0000","last_change":"2023-11-24T06:20:21.631414+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.631414+0000","last_undegraded":"2023-11-24T06:20:21.631414+0000","last_fullsized":"2023-11-24T06:20:21.631414+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639627+0000","last_change":"2023-11-24T06:20:21.634801+0000","last_active":"2023-11-24T06:20:21.639627+0000","last_peered":"2023-11-24T06:20:21.639627+0000","last_clean":"2023-11-24T06:20:21.639627+0000","last_became_active":"2023-11-24T06:20:21.634620+0000","last_became_peered":"2023-11-24T06:20:21.634620+0000","last_unstale":"2023-11-24T06:20:21.639627+0000","last_undegraded":"2023-11-24T06:20:21.639627+0000","last_fullsized":"2023-11-24T06:20:21.639627+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:26.240404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625721+0000","last_change":"2023-11-24T06:20:21.625721+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625721+0000","last_undegraded":"2023-11-24T06:20:21.625721+0000","last_fullsized":"2023-11-24T06:20:21.625721+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.647968+0000","last_change":"2023-11-24T06:20:21.641256+0000","last_active":"2023-11-24T06:20:21.647968+0000","last_peered":"2023-11-24T06:20:21.647968+0000","last_clean":"2023-11-24T06:20:21.647968+0000","last_became_active":"2023-11-24T06:20:21.640930+0000","last_became_peered":"2023-11-24T06:20:21.640930+0000","last_unstale":"2023-11-24T06:20:21.647968+0000","last_undegraded":"2023-11-24T06:20:21.647968+0000","last_fullsized":"2023-11-24T06:20:21.647968+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:38.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627331+0000","last_change":"2023-11-24T06:20:21.627331+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627331+0000","last_undegraded":"2023-11-24T06:20:21.627331+0000","last_fullsized":"2023-11-24T06:20:21.627331+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638358+0000","last_change":"2023-11-24T06:20:21.634844+0000","last_active":"2023-11-24T06:20:21.638358+0000","last_peered":"2023-11-24T06:20:21.638358+0000","last_clean":"2023-11-24T06:20:21.638358+0000","last_became_active":"2023-11-24T06:20:21.634564+0000","last_became_peered":"2023-11-24T06:20:21.634564+0000","last_unstale":"2023-11-24T06:20:21.638358+0000","last_undegraded":"2023-11-24T06:20:21.638358+0000","last_fullsized":"2023-11-24T06:20:21.638358+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623259+0000","last_change":"2023-11-24T06:20:21.623259+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623259+0000","last_undegraded":"2023-11-24T06:20:21.623259+0000","last_fullsized":"2023-11-24T06:20:21.623259+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.633149+0000","last_change":"2023-11-24T06:20:21.629599+0000","last_active":"2023-11-24T06:20:21.633149+0000","last_peered":"2023-11-24T06:20:21.633149+0000","last_clean":"2023-11-24T06:20:21.633149+0000","last_became_active":"2023-11-24T06:20:21.629219+0000","last_became_peered":"2023-11-24T06:20:21.629219+0000","last_unstale":"2023-11-24T06:20:21.633149+0000","last_undegraded":"2023-11-24T06:20:21.633149+0000","last_fullsized":"2023-11-24T06:20:21.633149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:18.021884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623607+0000","last_change":"2023-11-24T06:20:21.623607+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623607+0000","last_undegraded":"2023-11-24T06:20:21.623607+0000","last_fullsized":"2023-11-24T06:20:21.623607+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640257+0000","last_change":"2023-11-24T06:20:21.635521+0000","last_active":"2023-11-24T06:20:21.640257+0000","last_peered":"2023-11-24T06:20:21.640257+0000","last_clean":"2023-11-24T06:20:21.640257+0000","last_became_active":"2023-11-24T06:20:21.635214+0000","last_became_peered":"2023-11-24T06:20:21.635214+0000","last_unstale":"2023-11-24T06:20:21.640257+0000","last_undegraded":"2023-11-24T06:20:21.640257+0000","last_fullsized":"2023-11-24T06:20:21.640257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:23.435615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625448+0000","last_change":"2023-11-24T06:20:21.625448+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625448+0000","last_undegraded":"2023-11-24T06:20:21.625448+0000","last_fullsized":"2023-11-24T06:20:21.625448+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638387+0000","last_change":"2023-11-24T06:20:21.634192+0000","last_active":"2023-11-24T06:20:21.638387+0000","last_peered":"2023-11-24T06:20:21.638387+0000","last_clean":"2023-11-24T06:20:21.638387+0000","last_became_active":"2023-11-24T06:20:21.633818+0000","last_became_peered":"2023-11-24T06:20:21.633818+0000","last_unstale":"2023-11-24T06:20:21.638387+0000","last_undegraded":"2023-11-24T06:20:21.638387+0000","last_fullsized":"2023-11-24T06:20:21.638387+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:02:30.378149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624564+0000","last_change":"2023-11-24T06:20:21.624564+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624564+0000","last_undegraded":"2023-11-24T06:20:21.624564+0000","last_fullsized":"2023-11-24T06:20:21.624564+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640823+0000","last_change":"2023-11-24T06:20:21.637491+0000","last_active":"2023-11-24T06:20:21.640823+0000","last_peered":"2023-11-24T06:20:21.640823+0000","last_clean":"2023-11-24T06:20:21.640823+0000","last_became_active":"2023-11-24T06:20:21.637173+0000","last_became_peered":"2023-11-24T06:20:21.637173+0000","last_unstale":"2023-11-24T06:20:21.640823+0000","last_undegraded":"2023-11-24T06:20:21.640823+0000","last_fullsized":"2023-11-24T06:20:21.640823+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:15:39.025211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621408+0000","last_change":"2023-11-24T06:20:21.621408+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621408+0000","last_undegraded":"2023-11-24T06:20:21.621408+0000","last_fullsized":"2023-11-24T06:20:21.621408+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640898+0000","last_change":"2023-11-24T06:20:21.638447+0000","last_active":"2023-11-24T06:20:21.640898+0000","last_peered":"2023-11-24T06:20:21.640898+0000","last_clean":"2023-11-24T06:20:21.640898+0000","last_became_active":"2023-11-24T06:20:21.638158+0000","last_became_peered":"2023-11-24T06:20:21.638158+0000","last_unstale":"2023-11-24T06:20:21.640898+0000","last_undegraded":"2023-11-24T06:20:21.640898+0000","last_fullsized":"2023-11-24T06:20:21.640898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:51.335391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621969+0000","last_change":"2023-11-24T06:20:21.621969+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621969+0000","last_undegraded":"2023-11-24T06:20:21.621969+0000","last_fullsized":"2023-11-24T06:20:21.621969+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638472+0000","last_change":"2023-11-24T06:20:21.631051+0000","last_active":"2023-11-24T06:20:21.638472+0000","last_peered":"2023-11-24T06:20:21.638472+0000","last_clean":"2023-11-24T06:20:21.638472+0000","last_became_active":"2023-11-24T06:20:21.630773+0000","last_became_peered":"2023-11-24T06:20:21.630773+0000","last_unstale":"2023-11-24T06:20:21.638472+0000","last_undegraded":"2023-11-24T06:20:21.638472+0000","last_fullsized":"2023-11-24T06:20:21.638472+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:19.355871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627272+0000","last_change":"2023-11-24T06:20:21.627272+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627272+0000","last_undegraded":"2023-11-24T06:20:21.627272+0000","last_fullsized":"2023-11-24T06:20:21.627272+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640832+0000","last_change":"2023-11-24T06:20:21.637050+0000","last_active":"2023-11-24T06:20:21.640832+0000","last_peered":"2023-11-24T06:20:21.640832+0000","last_clean":"2023-11-24T06:20:21.640832+0000","last_became_active":"2023-11-24T06:20:21.636769+0000","last_became_peered":"2023-11-24T06:20:21.636769+0000","last_unstale":"2023-11-24T06:20:21.640832+0000","last_undegraded":"2023-11-24T06:20:21.640832+0000","last_fullsized":"2023-11-24T06:20:21.640832+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:41:45.468254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621600+0000","last_change":"2023-11-24T06:20:21.621600+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621600+0000","last_undegraded":"2023-11-24T06:20:21.621600+0000","last_fullsized":"2023-11-24T06:20:21.621600+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640858+0000","last_change":"2023-11-24T06:20:21.638561+0000","last_active":"2023-11-24T06:20:21.640858+0000","last_peered":"2023-11-24T06:20:21.640858+0000","last_clean":"2023-11-24T06:20:21.640858+0000","last_became_active":"2023-11-24T06:20:21.638252+0000","last_became_peered":"2023-11-24T06:20:21.638252+0000","last_unstale":"2023-11-24T06:20:21.640858+0000","last_undegraded":"2023-11-24T06:20:21.640858+0000","last_fullsized":"2023-11-24T06:20:21.640858+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:25.850351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627692+0000","last_change":"2023-11-24T06:20:21.627692+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627692+0000","last_undegraded":"2023-11-24T06:20:21.627692+0000","last_fullsized":"2023-11-24T06:20:21.627692+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639371+0000","last_change":"2023-11-24T06:20:21.633330+0000","last_active":"2023-11-24T06:20:21.639371+0000","last_peered":"2023-11-24T06:20:21.639371+0000","last_clean":"2023-11-24T06:20:21.639371+0000","last_became_active":"2023-11-24T06:20:21.633057+0000","last_became_peered":"2023-11-24T06:20:21.633057+0000","last_unstale":"2023-11-24T06:20:21.639371+0000","last_undegraded":"2023-11-24T06:20:21.639371+0000","last_fullsized":"2023-11-24T06:20:21.639371+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:29.077158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622661+0000","last_change":"2023-11-24T06:20:21.622661+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622661+0000","last_undegraded":"2023-11-24T06:20:21.622661+0000","last_fullsized":"2023-11-24T06:20:21.622661+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.633740+0000","last_change":"2023-11-24T06:20:21.632526+0000","last_active":"2023-11-24T06:20:21.633740+0000","last_peered":"2023-11-24T06:20:21.633740+0000","last_clean":"2023-11-24T06:20:21.633740+0000","last_became_active":"2023-11-24T06:20:21.632218+0000","last_became_peered":"2023-11-24T06:20:21.632218+0000","last_unstale":"2023-11-24T06:20:21.633740+0000","last_undegraded":"2023-11-24T06:20:21.633740+0000","last_fullsized":"2023-11-24T06:20:21.633740+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:57:23.439486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621256+0000","last_change":"2023-11-24T06:20:21.621256+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621256+0000","last_undegraded":"2023-11-24T06:20:21.621256+0000","last_fullsized":"2023-11-24T06:20:21.621256+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638406+0000","last_change":"2023-11-24T06:20:21.635534+0000","last_active":"2023-11-24T06:20:21.638406+0000","last_peered":"2023-11-24T06:20:21.638406+0000","last_clean":"2023-11-24T06:20:21.638406+0000","last_became_active":"2023-11-24T06:20:21.635099+0000","last_became_peered":"2023-11-24T06:20:21.635099+0000","last_unstale":"2023-11-24T06:20:21.638406+0000","last_undegraded":"2023-11-24T06:20:21.638406+0000","last_fullsized":"2023-11-24T06:20:21.638406+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:18:34.137206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621164+0000","last_change":"2023-11-24T06:20:21.621164+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621164+0000","last_undegraded":"2023-11-24T06:20:21.621164+0000","last_fullsized":"2023-11-24T06:20:21.621164+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.636688+0000","last_change":"2023-11-24T06:20:21.634951+0000","last_active":"2023-11-24T06:20:21.636688+0000","last_peered":"2023-11-24T06:20:21.636688+0000","last_clean":"2023-11-24T06:20:21.636688+0000","last_became_active":"2023-11-24T06:20:21.634653+0000","last_became_peered":"2023-11-24T06:20:21.634653+0000","last_unstale":"2023-11-24T06:20:21.636688+0000","last_undegraded":"2023-11-24T06:20:21.636688+0000","last_fullsized":"2023-11-24T06:20:21.636688+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:30.394419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621410+0000","last_change":"2023-11-24T06:20:21.621410+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621410+0000","last_undegraded":"2023-11-24T06:20:21.621410+0000","last_fullsized":"2023-11-24T06:20:21.621410+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639474+0000","last_change":"2023-11-24T06:20:21.633653+0000","last_active":"2023-11-24T06:20:21.639474+0000","last_peered":"2023-11-24T06:20:21.639474+0000","last_clean":"2023-11-24T06:20:21.639474+0000","last_became_active":"2023-11-24T06:20:21.633333+0000","last_became_peered":"2023-11-24T06:20:21.633333+0000","last_unstale":"2023-11-24T06:20:21.639474+0000","last_undegraded":"2023-11-24T06:20:21.639474+0000","last_fullsized":"2023-11-24T06:20:21.639474+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:44.134108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627180+0000","last_change":"2023-11-24T06:20:21.627180+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627180+0000","last_undegraded":"2023-11-24T06:20:21.627180+0000","last_fullsized":"2023-11-24T06:20:21.627180+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640570+0000","last_change":"2023-11-24T06:20:21.638281+0000","last_active":"2023-11-24T06:20:21.640570+0000","last_peered":"2023-11-24T06:20:21.640570+0000","last_clean":"2023-11-24T06:20:21.640570+0000","last_became_active":"2023-11-24T06:20:21.638014+0000","last_became_peered":"2023-11-24T06:20:21.638014+0000","last_unstale":"2023-11-24T06:20:21.640570+0000","last_undegraded":"2023-11-24T06:20:21.640570+0000","last_fullsized":"2023-11-24T06:20:21.640570+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:15:38.645013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623054+0000","last_change":"2023-11-24T06:20:21.623054+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623054+0000","last_undegraded":"2023-11-24T06:20:21.623054+0000","last_fullsized":"2023-11-24T06:20:21.623054+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638275+0000","last_change":"2023-11-24T06:20:21.634745+0000","last_active":"2023-11-24T06:20:21.638275+0000","last_peered":"2023-11-24T06:20:21.638275+0000","last_clean":"2023-11-24T06:20:21.638275+0000","last_became_active":"2023-11-24T06:20:21.634453+0000","last_became_peered":"2023-11-24T06:20:21.634453+0000","last_unstale":"2023-11-24T06:20:21.638275+0000","last_undegraded":"2023-11-24T06:20:21.638275+0000","last_fullsized":"2023-11-24T06:20:21.638275+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:24:57.040169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621052+0000","last_change":"2023-11-24T06:20:21.621052+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621052+0000","last_undegraded":"2023-11-24T06:20:21.621052+0000","last_fullsized":"2023-11-24T06:20:21.621052+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.642800+0000","last_change":"2023-11-24T06:20:21.640109+0000","last_active":"2023-11-24T06:20:21.642800+0000","last_peered":"2023-11-24T06:20:21.642800+0000","last_clean":"2023-11-24T06:20:21.642800+0000","last_became_active":"2023-11-24T06:20:21.639777+0000","last_became_peered":"2023-11-24T06:20:21.639777+0000","last_unstale":"2023-11-24T06:20:21.642800+0000","last_undegraded":"2023-11-24T06:20:21.642800+0000","last_fullsized":"2023-11-24T06:20:21.642800+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625137+0000","last_change":"2023-11-24T06:20:21.625137+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625137+0000","last_undegraded":"2023-11-24T06:20:21.625137+0000","last_fullsized":"2023-11-24T06:20:21.625137+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638300+0000","last_change":"2023-11-24T06:20:21.631857+0000","last_active":"2023-11-24T06:20:21.638300+0000","last_peered":"2023-11-24T06:20:21.638300+0000","last_clean":"2023-11-24T06:20:21.638300+0000","last_became_active":"2023-11-24T06:20:21.631396+0000","last_became_peered":"2023-11-24T06:20:21.631396+0000","last_unstale":"2023-11-24T06:20:21.638300+0000","last_undegraded":"2023-11-24T06:20:21.638300+0000","last_fullsized":"2023-11-24T06:20:21.638300+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:43.904787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625641+0000","last_change":"2023-11-24T06:20:21.625641+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625641+0000","last_undegraded":"2023-11-24T06:20:21.625641+0000","last_fullsized":"2023-11-24T06:20:21.625641+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.635183+0000","last_change":"2023-11-24T06:20:21.633654+0000","last_active":"2023-11-24T06:20:21.635183+0000","last_peered":"2023-11-24T06:20:21.635183+0000","last_clean":"2023-11-24T06:20:21.635183+0000","last_became_active":"2023-11-24T06:20:21.633387+0000","last_became_peered":"2023-11-24T06:20:21.633387+0000","last_unstale":"2023-11-24T06:20:21.635183+0000","last_undegraded":"2023-11-24T06:20:21.635183+0000","last_fullsized":"2023-11-24T06:20:21.635183+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:05.063980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625225+0000","last_change":"2023-11-24T06:20:21.625225+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625225+0000","last_undegraded":"2023-11-24T06:20:21.625225+0000","last_fullsized":"2023-11-24T06:20:21.625225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.641065+0000","last_change":"2023-11-24T06:20:21.639872+0000","last_active":"2023-11-24T06:20:21.641065+0000","last_peered":"2023-11-24T06:20:21.641065+0000","last_clean":"2023-11-24T06:20:21.641065+0000","last_became_active":"2023-11-24T06:20:21.639600+0000","last_became_peered":"2023-11-24T06:20:21.639600+0000","last_unstale":"2023-11-24T06:20:21.641065+0000","last_undegraded":"2023-11-24T06:20:21.641065+0000","last_fullsized":"2023-11-24T06:20:21.641065+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:15.519175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625436+0000","last_change":"2023-11-24T06:20:21.625436+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625436+0000","last_undegraded":"2023-11-24T06:20:21.625436+0000","last_fullsized":"2023-11-24T06:20:21.625436+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639363+0000","last_change":"2023-11-24T06:20:21.638221+0000","last_active":"2023-11-24T06:20:21.639363+0000","last_peered":"2023-11-24T06:20:21.639363+0000","last_clean":"2023-11-24T06:20:21.639363+0000","last_became_active":"2023-11-24T06:20:21.637800+0000","last_became_peered":"2023-11-24T06:20:21.637800+0000","last_unstale":"2023-11-24T06:20:21.639363+0000","last_undegraded":"2023-11-24T06:20:21.639363+0000","last_fullsized":"2023-11-24T06:20:21.639363+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:23.447147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.628285+0000","last_change":"2023-11-24T06:20:21.628285+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.628285+0000","last_undegraded":"2023-11-24T06:20:21.628285+0000","last_fullsized":"2023-11-24T06:20:21.628285+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640027+0000","last_change":"2023-11-24T06:20:21.635897+0000","last_active":"2023-11-24T06:20:21.640027+0000","last_peered":"2023-11-24T06:20:21.640027+0000","last_clean":"2023-11-24T06:20:21.640027+0000","last_became_active":"2023-11-24T06:20:21.635167+0000","last_became_peered":"2023-11-24T06:20:21.635167+0000","last_unstale":"2023-11-24T06:20:21.640027+0000","last_undegraded":"2023-11-24T06:20:21.640027+0000","last_fullsized":"2023-11-24T06:20:21.640027+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:54:05.115719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626755+0000","last_change":"2023-11-24T06:20:21.626755+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626755+0000","last_undegraded":"2023-11-24T06:20:21.626755+0000","last_fullsized":"2023-11-24T06:20:21.626755+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638367+0000","last_change":"2023-11-24T06:20:21.633952+0000","last_active":"2023-11-24T06:20:21.638367+0000","last_peered":"2023-11-24T06:20:21.638367+0000","last_clean":"2023-11-24T06:20:21.638367+0000","last_became_active":"2023-11-24T06:20:21.633629+0000","last_became_peered":"2023-11-24T06:20:21.633629+0000","last_unstale":"2023-11-24T06:20:21.638367+0000","last_undegraded":"2023-11-24T06:20:21.638367+0000","last_fullsized":"2023-11-24T06:20:21.638367+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:53.363443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626539+0000","last_change":"2023-11-24T06:20:21.626539+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626539+0000","last_undegraded":"2023-11-24T06:20:21.626539+0000","last_fullsized":"2023-11-24T06:20:21.626539+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638377+0000","last_change":"2023-11-24T06:20:21.634883+0000","last_active":"2023-11-24T06:20:21.638377+0000","last_peered":"2023-11-24T06:20:21.638377+0000","last_clean":"2023-11-24T06:20:21.638377+0000","last_became_active":"2023-11-24T06:20:21.634624+0000","last_became_peered":"2023-11-24T06:20:21.634624+0000","last_unstale":"2023-11-24T06:20:21.638377+0000","last_undegraded":"2023-11-24T06:20:21.638377+0000","last_fullsized":"2023-11-24T06:20:21.638377+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:38.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622724+0000","last_change":"2023-11-24T06:20:21.622724+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622724+0000","last_undegraded":"2023-11-24T06:20:21.622724+0000","last_fullsized":"2023-11-24T06:20:21.622724+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.637356+0000","last_change":"2023-11-24T06:20:21.635390+0000","last_active":"2023-11-24T06:20:21.637356+0000","last_peered":"2023-11-24T06:20:21.637356+0000","last_clean":"2023-11-24T06:20:21.637356+0000","last_became_active":"2023-11-24T06:20:21.635114+0000","last_became_peered":"2023-11-24T06:20:21.635114+0000","last_unstale":"2023-11-24T06:20:21.637356+0000","last_undegraded":"2023-11-24T06:20:21.637356+0000","last_fullsized":"2023-11-24T06:20:21.637356+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:15:39.025211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621069+0000","last_change":"2023-11-24T06:20:21.621069+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621069+0000","last_undegraded":"2023-11-24T06:20:21.621069+0000","last_fullsized":"2023-11-24T06:20:21.621069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640811+0000","last_change":"2023-11-24T06:20:21.636357+0000","last_active":"2023-11-24T06:20:21.640811+0000","last_peered":"2023-11-24T06:20:21.640811+0000","last_clean":"2023-11-24T06:20:21.640811+0000","last_became_active":"2023-11-24T06:20:21.636068+0000","last_became_peered":"2023-11-24T06:20:21.636068+0000","last_unstale":"2023-11-24T06:20:21.640811+0000","last_undegraded":"2023-11-24T06:20:21.640811+0000","last_fullsized":"2023-11-24T06:20:21.640811+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:24.432980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626644+0000","last_change":"2023-11-24T06:20:21.626644+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626644+0000","last_undegraded":"2023-11-24T06:20:21.626644+0000","last_fullsized":"2023-11-24T06:20:21.626644+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.641635+0000","last_change":"2023-11-24T06:20:21.641031+0000","last_active":"2023-11-24T06:20:21.641635+0000","last_peered":"2023-11-24T06:20:21.641635+0000","last_clean":"2023-11-24T06:20:21.641635+0000","last_became_active":"2023-11-24T06:20:21.640826+0000","last_became_peered":"2023-11-24T06:20:21.640826+0000","last_unstale":"2023-11-24T06:20:21.641635+0000","last_undegraded":"2023-11-24T06:20:21.641635+0000","last_fullsized":"2023-11-24T06:20:21.641635+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:20.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622735+0000","last_change":"2023-11-24T06:20:21.622735+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622735+0000","last_undegraded":"2023-11-24T06:20:21.622735+0000","last_fullsized":"2023-11-24T06:20:21.622735+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639612+0000","last_change":"2023-11-24T06:20:21.636646+0000","last_active":"2023-11-24T06:20:21.639612+0000","last_peered":"2023-11-24T06:20:21.639612+0000","last_clean":"2023-11-24T06:20:21.639612+0000","last_became_active":"2023-11-24T06:20:21.636335+0000","last_became_peered":"2023-11-24T06:20:21.636335+0000","last_unstale":"2023-11-24T06:20:21.639612+0000","last_undegraded":"2023-11-24T06:20:21.639612+0000","last_fullsized":"2023-11-24T06:20:21.639612+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623457+0000","last_change":"2023-11-24T06:20:21.623457+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623457+0000","last_undegraded":"2023-11-24T06:20:21.623457+0000","last_fullsized":"2023-11-24T06:20:21.623457+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638494+0000","last_change":"2023-11-24T06:20:21.635744+0000","last_active":"2023-11-24T06:20:21.638494+0000","last_peered":"2023-11-24T06:20:21.638494+0000","last_clean":"2023-11-24T06:20:21.638494+0000","last_became_active":"2023-11-24T06:20:21.635447+0000","last_became_peered":"2023-11-24T06:20:21.635447+0000","last_unstale":"2023-11-24T06:20:21.638494+0000","last_undegraded":"2023-11-24T06:20:21.638494+0000","last_fullsized":"2023-11-24T06:20:21.638494+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:17.983184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-11-24T06:20:20.618880+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:20.618880+0000","last_peered":"2023-11-24T06:20:20.618880+0000","last_clean":"2023-11-24T06:20:20.618880+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:20.618880+0000","last_undegraded":"2023-11-24T06:20:20.618880+0000","last_fullsized":"2023-11-24T06:20:20.618880+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":841444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.70899999999999996}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.81000000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.78000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":6.423,"5min":6.423,"15min":6.423},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":12.554,"5min":12.554,"15min":12.554},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.75700000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.81599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":257343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":841444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.255}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.30599999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.373}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.61199999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56200000000000006}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.57299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.41799999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.37},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.58799999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.89200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.223}]},{"osd":1,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.246},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.28899999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.317},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.30199999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.434},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53500000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.627},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67500000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.66200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.04,"5min":3.04,"15min":3.04},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":10.198,"5min":10.198,"15min":10.198},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":10.215999999999999,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.63500000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.66000000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:20:27.136 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:20:27.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.289+0000 7facfac5d700 1 Processor -- start 2023-11-24T06:20:27.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.289+0000 7facfac5d700 1 -- start start 2023-11-24T06:20:27.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facfac5d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415dfc0 0x7facf40d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facfac5d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7facf40b2fc0 0x7facf40d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facfac5d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf40d2d10 con 0x7facf40b2a70 2023-11-24T06:20:27.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facfac5d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf40cefd0 con 0x7facf40b2fc0 2023-11-24T06:20:27.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facfac5d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf40cf2b0 con 0x7facf415dfc0 2023-11-24T06:20:27.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf91fa700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7facf40b2a70 0x7facf4161b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41496/0 (socket says 172.21.15.89:41496) 2023-11-24T06:20:27.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf91fa700 1 -- 172.21.15.89:0/1514334538 learned_addr learned my addr 172.21.15.89:0/1514334538 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:27.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473825082 0 0) 0x7facf40d2d10 con 0x7facf40b2a70 2023-11-24T06:20:27.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face40035b0 con 0x7facf40b2a70 2023-11-24T06:20:27.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf3fff700 1 --2- 172.21.15.89:0/1514334538 >> v2:172.21.15.123:3300/0 conn(0x7facf40b2fc0 0x7facf40d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf89f9700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415dfc0 0x7facf40d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197705958 0 0) 0x7face40035b0 con 0x7facf40b2a70 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415dfc0 msgr2=0x7facf40d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415dfc0 0x7facf40d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 >> v2:172.21.15.123:3300/0 conn(0x7facf40b2fc0 msgr2=0x7facf40d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 --2- 172.21.15.89:0/1514334538 >> v2:172.21.15.123:3300/0 conn(0x7facf40b2fc0 0x7facf40d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40d2fb0 con 0x7facf40b2a70 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7facec0019f0 con 0x7facf40b2a70 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 >> v1:172.21.15.89:6789/0 conn(0x7facf40b2a70 legacy=0x7facf4161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.290+0000 7facf37fe700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415dfc0 0x7face4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf37fe700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7face4003f80 0x7face4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf37fe700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face4004d90 0x7face4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40cefd0 con 0x7face4004d90 2023-11-24T06:20:27.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40cf2b0 con 0x7face4003f80 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face400ba60 con 0x7facf415dfc0 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf91fa700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face4004d90 0x7face4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf89f9700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415dfc0 0x7face4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf3fff700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7face4003f80 0x7face4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf91fa700 1 -- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415dfc0 msgr2=0x7face4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf91fa700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415dfc0 0x7face4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf91fa700 1 -- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7face4003f80 msgr2=0x7face4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf91fa700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7face4003f80 0x7face4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf91fa700 1 -- 172.21.15.89:0/1514334538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face40035b0 con 0x7face4004d90 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf91fa700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face4004d90 0x7face4005210 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7facec00a900 tx=0x7face4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce178f872144f5e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec00a340 con 0x7face4004d90 2023-11-24T06:20:27.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec012620 con 0x7face4004d90 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.291+0000 7facf37fe700 1 -- 172.21.15.89:0/1514334538 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec0129d0 con 0x7face4004d90 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.292+0000 7facfac5d700 1 -- 172.21.15.89:0/1514334538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face4004d90 msgr2=0x7face4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.292+0000 7facfac5d700 1 --2- 172.21.15.89:0/1514334538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face4004d90 0x7face4005210 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7facec00a900 tx=0x7face4005880 comp rx=0 tx=0).stop 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.292+0000 7facf91fa700 1 -- 172.21.15.89:0/1514334538 reap_dead start 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.292+0000 7facfac5d700 1 -- 172.21.15.89:0/1514334538 shutdown_connections 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.292+0000 7facfac5d700 1 -- 172.21.15.89:0/1514334538 >> 172.21.15.89:0/1514334538 conn(0x7facf406b410 msgr2=0x7facf406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.292+0000 7facfac5d700 1 -- 172.21.15.89:0/1514334538 shutdown_connections 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.292+0000 7facfac5d700 1 -- 172.21.15.89:0/1514334538 wait complete. 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.293+0000 7facfac5d700 1 Processor -- start 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.293+0000 7facfac5d700 1 -- start start 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facfac5d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415ca20 0x7facf41f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facfac5d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facf415d300 0x7facf40c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facfac5d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facf41f4030 0x7facf40c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facfac5d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf41f4c90 con 0x7facf415d300 2023-11-24T06:20:27.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facfac5d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf41f4ae0 con 0x7facf41f4030 2023-11-24T06:20:27.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facfac5d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf40cc760 con 0x7facf415ca20 2023-11-24T06:20:27.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facf415d300 0x7facf40c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facf415d300 0x7facf40c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38002/0 (socket says 172.21.15.89:38002) 2023-11-24T06:20:27.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 -- 172.21.15.89:0/4215349036 learned_addr learned my addr 172.21.15.89:0/4215349036 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:27.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf91fa700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facf41f4030 0x7facf40c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf89f9700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415ca20 0x7facf41f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 -- 172.21.15.89:0/4215349036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415ca20 msgr2=0x7facf41f3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415ca20 0x7facf41f3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 -- 172.21.15.89:0/4215349036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facf41f4030 msgr2=0x7facf40c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facf41f4030 0x7facf40c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 -- 172.21.15.89:0/4215349036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf41f4520 con 0x7facf415d300 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf3fff700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facf415d300 0x7facf40c8f90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7face0011aa0 tx=0x7face0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.294+0000 7facf17fa700 1 -- 172.21.15.89:0/4215349036 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face000f9e0 con 0x7facf415d300 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.295+0000 7facf17fa700 1 -- 172.21.15.89:0/4215349036 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face000fb40 con 0x7facf415d300 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.295+0000 7facf17fa700 1 -- 172.21.15.89:0/4215349036 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0014440 con 0x7facf415d300 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.295+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf41f4800 con 0x7facf415d300 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.295+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf415ce00 con 0x7facf415d300 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.296+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf415d020 con 0x7facf415d300 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.296+0000 7facf17fa700 1 -- 172.21.15.89:0/4215349036 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7face00145a0 con 0x7facf415d300 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.296+0000 7facf17fa700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7facd806aed0 0x7facd806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.296+0000 7facf17fa700 1 -- 172.21.15.89:0/4215349036 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7face0089dd0 con 0x7facf415d300 2023-11-24T06:20:27.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.296+0000 7facf89f9700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7facd806aed0 0x7facd806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.297+0000 7facf89f9700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7facd806aed0 0x7facd806d390 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7facf41e84f0 tx=0x7face8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:27.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.301+0000 7facf17fa700 1 -- 172.21.15.89:0/4215349036 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facf415d020 con 0x7facf415d300 2023-11-24T06:20:27.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.487+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facf40d1aa0 con 0x7facd806aed0 2023-11-24T06:20:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.495+0000 7facf17fa700 1 -- 172.21.15.89:0/4215349036 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284268 (secure 0 0 0) 0x7facf40d1aa0 con 0x7facd806aed0 2023-11-24T06:20:27.501 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:27.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7facd806aed0 msgr2=0x7facd806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7facd806aed0 0x7facd806d390 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7facf41e84f0 tx=0x7face8008040 comp rx=0 tx=0).stop 2023-11-24T06:20:27.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facf415d300 msgr2=0x7facf40c8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facf415d300 0x7facf40c8f90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7face0011aa0 tx=0x7face0011e60 comp rx=0 tx=0).stop 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 shutdown_connections 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7facd806aed0 0x7facd806d390 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facf41f4030 0x7facf40c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7facf415ca20 0x7facf41f3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 --2- 172.21.15.89:0/4215349036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facf415d300 0x7facf40c8f90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 >> 172.21.15.89:0/4215349036 conn(0x7facf406b410 msgr2=0x7facf40b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 shutdown_connections 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.498+0000 7facfac5d700 1 -- 172.21.15.89:0/4215349036 wait complete. 2023-11-24T06:20:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:20:27.513 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2023-11-24T06:20:25.738893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222024,"kb_used_data":5844,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752696,"statfs":{"total":773094113280,"available":772866760704,"internally_reserved":0,"allocated":5984256,"data_stored":3195611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.123598"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623555+0000","last_change":"2023-11-24T06:20:21.623555+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623555+0000","last_undegraded":"2023-11-24T06:20:21.623555+0000","last_fullsized":"2023-11-24T06:20:21.623555+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.628738+0000","last_change":"2023-11-24T06:20:21.628738+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.628738+0000","last_undegraded":"2023-11-24T06:20:21.628738+0000","last_fullsized":"2023-11-24T06:20:21.628738+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623537+0000","last_change":"2023-11-24T06:20:21.623537+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623537+0000","last_undegraded":"2023-11-24T06:20:21.623537+0000","last_fullsized":"2023-11-24T06:20:21.623537+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626348+0000","last_change":"2023-11-24T06:20:21.626348+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626348+0000","last_undegraded":"2023-11-24T06:20:21.626348+0000","last_fullsized":"2023-11-24T06:20:21.626348+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624458+0000","last_change":"2023-11-24T06:20:21.624458+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624458+0000","last_undegraded":"2023-11-24T06:20:21.624458+0000","last_fullsized":"2023-11-24T06:20:21.624458+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627324+0000","last_change":"2023-11-24T06:20:21.627324+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627324+0000","last_undegraded":"2023-11-24T06:20:21.627324+0000","last_fullsized":"2023-11-24T06:20:21.627324+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622653+0000","last_change":"2023-11-24T06:20:21.622653+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622653+0000","last_undegraded":"2023-11-24T06:20:21.622653+0000","last_fullsized":"2023-11-24T06:20:21.622653+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.629361+0000","last_change":"2023-11-24T06:20:21.629361+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.629361+0000","last_undegraded":"2023-11-24T06:20:21.629361+0000","last_fullsized":"2023-11-24T06:20:21.629361+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.620894+0000","last_change":"2023-11-24T06:20:21.620894+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.620894+0000","last_undegraded":"2023-11-24T06:20:21.620894+0000","last_fullsized":"2023-11-24T06:20:21.620894+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626826+0000","last_change":"2023-11-24T06:20:21.626826+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626826+0000","last_undegraded":"2023-11-24T06:20:21.626826+0000","last_fullsized":"2023-11-24T06:20:21.626826+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.630536+0000","last_change":"2023-11-24T06:20:21.630536+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.630536+0000","last_undegraded":"2023-11-24T06:20:21.630536+0000","last_fullsized":"2023-11-24T06:20:21.630536+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625573+0000","last_change":"2023-11-24T06:20:21.625573+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625573+0000","last_undegraded":"2023-11-24T06:20:21.625573+0000","last_fullsized":"2023-11-24T06:20:21.625573+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626066+0000","last_change":"2023-11-24T06:20:21.626066+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626066+0000","last_undegraded":"2023-11-24T06:20:21.626066+0000","last_fullsized":"2023-11-24T06:20:21.626066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622517+0000","last_change":"2023-11-24T06:20:21.622517+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622517+0000","last_undegraded":"2023-11-24T06:20:21.622517+0000","last_fullsized":"2023-11-24T06:20:21.622517+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.620882+0000","last_change":"2023-11-24T06:20:21.620882+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.620882+0000","last_undegraded":"2023-11-24T06:20:21.620882+0000","last_fullsized":"2023-11-24T06:20:21.620882+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624061+0000","last_change":"2023-11-24T06:20:21.624061+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624061+0000","last_undegraded":"2023-11-24T06:20:21.624061+0000","last_fullsized":"2023-11-24T06:20:21.624061+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626805+0000","last_change":"2023-11-24T06:20:21.626805+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626805+0000","last_undegraded":"2023-11-24T06:20:21.626805+0000","last_fullsized":"2023-11-24T06:20:21.626805+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623770+0000","last_change":"2023-11-24T06:20:21.623770+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623770+0000","last_undegraded":"2023-11-24T06:20:21.623770+0000","last_fullsized":"2023-11-24T06:20:21.623770+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624008+0000","last_change":"2023-11-24T06:20:21.624008+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624008+0000","last_undegraded":"2023-11-24T06:20:21.624008+0000","last_fullsized":"2023-11-24T06:20:21.624008+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625942+0000","last_change":"2023-11-24T06:20:21.625942+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625942+0000","last_undegraded":"2023-11-24T06:20:21.625942+0000","last_fullsized":"2023-11-24T06:20:21.625942+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623073+0000","last_change":"2023-11-24T06:20:21.623073+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623073+0000","last_undegraded":"2023-11-24T06:20:21.623073+0000","last_fullsized":"2023-11-24T06:20:21.623073+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625109+0000","last_change":"2023-11-24T06:20:21.625109+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625109+0000","last_undegraded":"2023-11-24T06:20:21.625109+0000","last_fullsized":"2023-11-24T06:20:21.625109+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622512+0000","last_change":"2023-11-24T06:20:21.622512+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622512+0000","last_undegraded":"2023-11-24T06:20:21.622512+0000","last_fullsized":"2023-11-24T06:20:21.622512+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627183+0000","last_change":"2023-11-24T06:20:21.627183+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627183+0000","last_undegraded":"2023-11-24T06:20:21.627183+0000","last_fullsized":"2023-11-24T06:20:21.627183+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623106+0000","last_change":"2023-11-24T06:20:21.623106+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623106+0000","last_undegraded":"2023-11-24T06:20:21.623106+0000","last_fullsized":"2023-11-24T06:20:21.623106+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625894+0000","last_change":"2023-11-24T06:20:21.625894+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625894+0000","last_undegraded":"2023-11-24T06:20:21.625894+0000","last_fullsized":"2023-11-24T06:20:21.625894+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627857+0000","last_change":"2023-11-24T06:20:21.627857+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627857+0000","last_undegraded":"2023-11-24T06:20:21.627857+0000","last_fullsized":"2023-11-24T06:20:21.627857+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622254+0000","last_change":"2023-11-24T06:20:21.622254+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622254+0000","last_undegraded":"2023-11-24T06:20:21.622254+0000","last_fullsized":"2023-11-24T06:20:21.622254+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625015+0000","last_change":"2023-11-24T06:20:21.625015+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625015+0000","last_undegraded":"2023-11-24T06:20:21.625015+0000","last_fullsized":"2023-11-24T06:20:21.625015+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621840+0000","last_change":"2023-11-24T06:20:21.621840+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621840+0000","last_undegraded":"2023-11-24T06:20:21.621840+0000","last_fullsized":"2023-11-24T06:20:21.621840+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621760+0000","last_change":"2023-11-24T06:20:21.621760+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621760+0000","last_undegraded":"2023-11-24T06:20:21.621760+0000","last_fullsized":"2023-11-24T06:20:21.621760+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624542+0000","last_change":"2023-11-24T06:20:21.624542+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624542+0000","last_undegraded":"2023-11-24T06:20:21.624542+0000","last_fullsized":"2023-11-24T06:20:21.624542+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626103+0000","last_change":"2023-11-24T06:20:21.626103+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626103+0000","last_undegraded":"2023-11-24T06:20:21.626103+0000","last_fullsized":"2023-11-24T06:20:21.626103+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640218+0000","last_change":"2023-11-24T06:20:21.635404+0000","last_active":"2023-11-24T06:20:21.640218+0000","last_peered":"2023-11-24T06:20:21.640218+0000","last_clean":"2023-11-24T06:20:21.640218+0000","last_became_active":"2023-11-24T06:20:21.635136+0000","last_became_peered":"2023-11-24T06:20:21.635136+0000","last_unstale":"2023-11-24T06:20:21.640218+0000","last_undegraded":"2023-11-24T06:20:21.640218+0000","last_fullsized":"2023-11-24T06:20:21.640218+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:32.580874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.631414+0000","last_change":"2023-11-24T06:20:21.631414+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.631414+0000","last_undegraded":"2023-11-24T06:20:21.631414+0000","last_fullsized":"2023-11-24T06:20:21.631414+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639627+0000","last_change":"2023-11-24T06:20:21.634801+0000","last_active":"2023-11-24T06:20:21.639627+0000","last_peered":"2023-11-24T06:20:21.639627+0000","last_clean":"2023-11-24T06:20:21.639627+0000","last_became_active":"2023-11-24T06:20:21.634620+0000","last_became_peered":"2023-11-24T06:20:21.634620+0000","last_unstale":"2023-11-24T06:20:21.639627+0000","last_undegraded":"2023-11-24T06:20:21.639627+0000","last_fullsized":"2023-11-24T06:20:21.639627+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:26.240404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625721+0000","last_change":"2023-11-24T06:20:21.625721+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625721+0000","last_undegraded":"2023-11-24T06:20:21.625721+0000","last_fullsized":"2023-11-24T06:20:21.625721+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.647968+0000","last_change":"2023-11-24T06:20:21.641256+0000","last_active":"2023-11-24T06:20:21.647968+0000","last_peered":"2023-11-24T06:20:21.647968+0000","last_clean":"2023-11-24T06:20:21.647968+0000","last_became_active":"2023-11-24T06:20:21.640930+0000","last_became_peered":"2023-11-24T06:20:21.640930+0000","last_unstale":"2023-11-24T06:20:21.647968+0000","last_undegraded":"2023-11-24T06:20:21.647968+0000","last_fullsized":"2023-11-24T06:20:21.647968+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:38.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627331+0000","last_change":"2023-11-24T06:20:21.627331+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627331+0000","last_undegraded":"2023-11-24T06:20:21.627331+0000","last_fullsized":"2023-11-24T06:20:21.627331+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638358+0000","last_change":"2023-11-24T06:20:21.634844+0000","last_active":"2023-11-24T06:20:21.638358+0000","last_peered":"2023-11-24T06:20:21.638358+0000","last_clean":"2023-11-24T06:20:21.638358+0000","last_became_active":"2023-11-24T06:20:21.634564+0000","last_became_peered":"2023-11-24T06:20:21.634564+0000","last_unstale":"2023-11-24T06:20:21.638358+0000","last_undegraded":"2023-11-24T06:20:21.638358+0000","last_fullsized":"2023-11-24T06:20:21.638358+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623259+0000","last_change":"2023-11-24T06:20:21.623259+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623259+0000","last_undegraded":"2023-11-24T06:20:21.623259+0000","last_fullsized":"2023-11-24T06:20:21.623259+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.633149+0000","last_change":"2023-11-24T06:20:21.629599+0000","last_active":"2023-11-24T06:20:21.633149+0000","last_peered":"2023-11-24T06:20:21.633149+0000","last_clean":"2023-11-24T06:20:21.633149+0000","last_became_active":"2023-11-24T06:20:21.629219+0000","last_became_peered":"2023-11-24T06:20:21.629219+0000","last_unstale":"2023-11-24T06:20:21.633149+0000","last_undegraded":"2023-11-24T06:20:21.633149+0000","last_fullsized":"2023-11-24T06:20:21.633149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:00:18.021884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623607+0000","last_change":"2023-11-24T06:20:21.623607+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623607+0000","last_undegraded":"2023-11-24T06:20:21.623607+0000","last_fullsized":"2023-11-24T06:20:21.623607+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640257+0000","last_change":"2023-11-24T06:20:21.635521+0000","last_active":"2023-11-24T06:20:21.640257+0000","last_peered":"2023-11-24T06:20:21.640257+0000","last_clean":"2023-11-24T06:20:21.640257+0000","last_became_active":"2023-11-24T06:20:21.635214+0000","last_became_peered":"2023-11-24T06:20:21.635214+0000","last_unstale":"2023-11-24T06:20:21.640257+0000","last_undegraded":"2023-11-24T06:20:21.640257+0000","last_fullsized":"2023-11-24T06:20:21.640257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:23.435615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625448+0000","last_change":"2023-11-24T06:20:21.625448+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625448+0000","last_undegraded":"2023-11-24T06:20:21.625448+0000","last_fullsized":"2023-11-24T06:20:21.625448+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638387+0000","last_change":"2023-11-24T06:20:21.634192+0000","last_active":"2023-11-24T06:20:21.638387+0000","last_peered":"2023-11-24T06:20:21.638387+0000","last_clean":"2023-11-24T06:20:21.638387+0000","last_became_active":"2023-11-24T06:20:21.633818+0000","last_became_peered":"2023-11-24T06:20:21.633818+0000","last_unstale":"2023-11-24T06:20:21.638387+0000","last_undegraded":"2023-11-24T06:20:21.638387+0000","last_fullsized":"2023-11-24T06:20:21.638387+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:02:30.378149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.624564+0000","last_change":"2023-11-24T06:20:21.624564+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.624564+0000","last_undegraded":"2023-11-24T06:20:21.624564+0000","last_fullsized":"2023-11-24T06:20:21.624564+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640823+0000","last_change":"2023-11-24T06:20:21.637491+0000","last_active":"2023-11-24T06:20:21.640823+0000","last_peered":"2023-11-24T06:20:21.640823+0000","last_clean":"2023-11-24T06:20:21.640823+0000","last_became_active":"2023-11-24T06:20:21.637173+0000","last_became_peered":"2023-11-24T06:20:21.637173+0000","last_unstale":"2023-11-24T06:20:21.640823+0000","last_undegraded":"2023-11-24T06:20:21.640823+0000","last_fullsized":"2023-11-24T06:20:21.640823+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:15:39.025211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621408+0000","last_change":"2023-11-24T06:20:21.621408+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621408+0000","last_undegraded":"2023-11-24T06:20:21.621408+0000","last_fullsized":"2023-11-24T06:20:21.621408+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640898+0000","last_change":"2023-11-24T06:20:21.638447+0000","last_active":"2023-11-24T06:20:21.640898+0000","last_peered":"2023-11-24T06:20:21.640898+0000","last_clean":"2023-11-24T06:20:21.640898+0000","last_became_active":"2023-11-24T06:20:21.638158+0000","last_became_peered":"2023-11-24T06:20:21.638158+0000","last_unstale":"2023-11-24T06:20:21.640898+0000","last_undegraded":"2023-11-24T06:20:21.640898+0000","last_fullsized":"2023-11-24T06:20:21.640898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:51.335391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621969+0000","last_change":"2023-11-24T06:20:21.621969+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621969+0000","last_undegraded":"2023-11-24T06:20:21.621969+0000","last_fullsized":"2023-11-24T06:20:21.621969+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638472+0000","last_change":"2023-11-24T06:20:21.631051+0000","last_active":"2023-11-24T06:20:21.638472+0000","last_peered":"2023-11-24T06:20:21.638472+0000","last_clean":"2023-11-24T06:20:21.638472+0000","last_became_active":"2023-11-24T06:20:21.630773+0000","last_became_peered":"2023-11-24T06:20:21.630773+0000","last_unstale":"2023-11-24T06:20:21.638472+0000","last_undegraded":"2023-11-24T06:20:21.638472+0000","last_fullsized":"2023-11-24T06:20:21.638472+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:19.355871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627272+0000","last_change":"2023-11-24T06:20:21.627272+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627272+0000","last_undegraded":"2023-11-24T06:20:21.627272+0000","last_fullsized":"2023-11-24T06:20:21.627272+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640832+0000","last_change":"2023-11-24T06:20:21.637050+0000","last_active":"2023-11-24T06:20:21.640832+0000","last_peered":"2023-11-24T06:20:21.640832+0000","last_clean":"2023-11-24T06:20:21.640832+0000","last_became_active":"2023-11-24T06:20:21.636769+0000","last_became_peered":"2023-11-24T06:20:21.636769+0000","last_unstale":"2023-11-24T06:20:21.640832+0000","last_undegraded":"2023-11-24T06:20:21.640832+0000","last_fullsized":"2023-11-24T06:20:21.640832+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:41:45.468254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621600+0000","last_change":"2023-11-24T06:20:21.621600+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621600+0000","last_undegraded":"2023-11-24T06:20:21.621600+0000","last_fullsized":"2023-11-24T06:20:21.621600+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640858+0000","last_change":"2023-11-24T06:20:21.638561+0000","last_active":"2023-11-24T06:20:21.640858+0000","last_peered":"2023-11-24T06:20:21.640858+0000","last_clean":"2023-11-24T06:20:21.640858+0000","last_became_active":"2023-11-24T06:20:21.638252+0000","last_became_peered":"2023-11-24T06:20:21.638252+0000","last_unstale":"2023-11-24T06:20:21.640858+0000","last_undegraded":"2023-11-24T06:20:21.640858+0000","last_fullsized":"2023-11-24T06:20:21.640858+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:25.850351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627692+0000","last_change":"2023-11-24T06:20:21.627692+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627692+0000","last_undegraded":"2023-11-24T06:20:21.627692+0000","last_fullsized":"2023-11-24T06:20:21.627692+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639371+0000","last_change":"2023-11-24T06:20:21.633330+0000","last_active":"2023-11-24T06:20:21.639371+0000","last_peered":"2023-11-24T06:20:21.639371+0000","last_clean":"2023-11-24T06:20:21.639371+0000","last_became_active":"2023-11-24T06:20:21.633057+0000","last_became_peered":"2023-11-24T06:20:21.633057+0000","last_unstale":"2023-11-24T06:20:21.639371+0000","last_undegraded":"2023-11-24T06:20:21.639371+0000","last_fullsized":"2023-11-24T06:20:21.639371+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:29.077158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622661+0000","last_change":"2023-11-24T06:20:21.622661+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622661+0000","last_undegraded":"2023-11-24T06:20:21.622661+0000","last_fullsized":"2023-11-24T06:20:21.622661+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.633740+0000","last_change":"2023-11-24T06:20:21.632526+0000","last_active":"2023-11-24T06:20:21.633740+0000","last_peered":"2023-11-24T06:20:21.633740+0000","last_clean":"2023-11-24T06:20:21.633740+0000","last_became_active":"2023-11-24T06:20:21.632218+0000","last_became_peered":"2023-11-24T06:20:21.632218+0000","last_unstale":"2023-11-24T06:20:21.633740+0000","last_undegraded":"2023-11-24T06:20:21.633740+0000","last_fullsized":"2023-11-24T06:20:21.633740+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:57:23.439486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621256+0000","last_change":"2023-11-24T06:20:21.621256+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621256+0000","last_undegraded":"2023-11-24T06:20:21.621256+0000","last_fullsized":"2023-11-24T06:20:21.621256+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638406+0000","last_change":"2023-11-24T06:20:21.635534+0000","last_active":"2023-11-24T06:20:21.638406+0000","last_peered":"2023-11-24T06:20:21.638406+0000","last_clean":"2023-11-24T06:20:21.638406+0000","last_became_active":"2023-11-24T06:20:21.635099+0000","last_became_peered":"2023-11-24T06:20:21.635099+0000","last_unstale":"2023-11-24T06:20:21.638406+0000","last_undegraded":"2023-11-24T06:20:21.638406+0000","last_fullsized":"2023-11-24T06:20:21.638406+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:18:34.137206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621164+0000","last_change":"2023-11-24T06:20:21.621164+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621164+0000","last_undegraded":"2023-11-24T06:20:21.621164+0000","last_fullsized":"2023-11-24T06:20:21.621164+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.636688+0000","last_change":"2023-11-24T06:20:21.634951+0000","last_active":"2023-11-24T06:20:21.636688+0000","last_peered":"2023-11-24T06:20:21.636688+0000","last_clean":"2023-11-24T06:20:21.636688+0000","last_became_active":"2023-11-24T06:20:21.634653+0000","last_became_peered":"2023-11-24T06:20:21.634653+0000","last_unstale":"2023-11-24T06:20:21.636688+0000","last_undegraded":"2023-11-24T06:20:21.636688+0000","last_fullsized":"2023-11-24T06:20:21.636688+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:30.394419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621410+0000","last_change":"2023-11-24T06:20:21.621410+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621410+0000","last_undegraded":"2023-11-24T06:20:21.621410+0000","last_fullsized":"2023-11-24T06:20:21.621410+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639474+0000","last_change":"2023-11-24T06:20:21.633653+0000","last_active":"2023-11-24T06:20:21.639474+0000","last_peered":"2023-11-24T06:20:21.639474+0000","last_clean":"2023-11-24T06:20:21.639474+0000","last_became_active":"2023-11-24T06:20:21.633333+0000","last_became_peered":"2023-11-24T06:20:21.633333+0000","last_unstale":"2023-11-24T06:20:21.639474+0000","last_undegraded":"2023-11-24T06:20:21.639474+0000","last_fullsized":"2023-11-24T06:20:21.639474+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:44.134108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.627180+0000","last_change":"2023-11-24T06:20:21.627180+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.627180+0000","last_undegraded":"2023-11-24T06:20:21.627180+0000","last_fullsized":"2023-11-24T06:20:21.627180+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640570+0000","last_change":"2023-11-24T06:20:21.638281+0000","last_active":"2023-11-24T06:20:21.640570+0000","last_peered":"2023-11-24T06:20:21.640570+0000","last_clean":"2023-11-24T06:20:21.640570+0000","last_became_active":"2023-11-24T06:20:21.638014+0000","last_became_peered":"2023-11-24T06:20:21.638014+0000","last_unstale":"2023-11-24T06:20:21.640570+0000","last_undegraded":"2023-11-24T06:20:21.640570+0000","last_fullsized":"2023-11-24T06:20:21.640570+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:15:38.645013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623054+0000","last_change":"2023-11-24T06:20:21.623054+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623054+0000","last_undegraded":"2023-11-24T06:20:21.623054+0000","last_fullsized":"2023-11-24T06:20:21.623054+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638275+0000","last_change":"2023-11-24T06:20:21.634745+0000","last_active":"2023-11-24T06:20:21.638275+0000","last_peered":"2023-11-24T06:20:21.638275+0000","last_clean":"2023-11-24T06:20:21.638275+0000","last_became_active":"2023-11-24T06:20:21.634453+0000","last_became_peered":"2023-11-24T06:20:21.634453+0000","last_unstale":"2023-11-24T06:20:21.638275+0000","last_undegraded":"2023-11-24T06:20:21.638275+0000","last_fullsized":"2023-11-24T06:20:21.638275+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:24:57.040169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621052+0000","last_change":"2023-11-24T06:20:21.621052+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621052+0000","last_undegraded":"2023-11-24T06:20:21.621052+0000","last_fullsized":"2023-11-24T06:20:21.621052+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.642800+0000","last_change":"2023-11-24T06:20:21.640109+0000","last_active":"2023-11-24T06:20:21.642800+0000","last_peered":"2023-11-24T06:20:21.642800+0000","last_clean":"2023-11-24T06:20:21.642800+0000","last_became_active":"2023-11-24T06:20:21.639777+0000","last_became_peered":"2023-11-24T06:20:21.639777+0000","last_unstale":"2023-11-24T06:20:21.642800+0000","last_undegraded":"2023-11-24T06:20:21.642800+0000","last_fullsized":"2023-11-24T06:20:21.642800+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625137+0000","last_change":"2023-11-24T06:20:21.625137+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625137+0000","last_undegraded":"2023-11-24T06:20:21.625137+0000","last_fullsized":"2023-11-24T06:20:21.625137+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638300+0000","last_change":"2023-11-24T06:20:21.631857+0000","last_active":"2023-11-24T06:20:21.638300+0000","last_peered":"2023-11-24T06:20:21.638300+0000","last_clean":"2023-11-24T06:20:21.638300+0000","last_became_active":"2023-11-24T06:20:21.631396+0000","last_became_peered":"2023-11-24T06:20:21.631396+0000","last_unstale":"2023-11-24T06:20:21.638300+0000","last_undegraded":"2023-11-24T06:20:21.638300+0000","last_fullsized":"2023-11-24T06:20:21.638300+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:43.904787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625641+0000","last_change":"2023-11-24T06:20:21.625641+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625641+0000","last_undegraded":"2023-11-24T06:20:21.625641+0000","last_fullsized":"2023-11-24T06:20:21.625641+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.635183+0000","last_change":"2023-11-24T06:20:21.633654+0000","last_active":"2023-11-24T06:20:21.635183+0000","last_peered":"2023-11-24T06:20:21.635183+0000","last_clean":"2023-11-24T06:20:21.635183+0000","last_became_active":"2023-11-24T06:20:21.633387+0000","last_became_peered":"2023-11-24T06:20:21.633387+0000","last_unstale":"2023-11-24T06:20:21.635183+0000","last_undegraded":"2023-11-24T06:20:21.635183+0000","last_fullsized":"2023-11-24T06:20:21.635183+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:05.063980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625225+0000","last_change":"2023-11-24T06:20:21.625225+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625225+0000","last_undegraded":"2023-11-24T06:20:21.625225+0000","last_fullsized":"2023-11-24T06:20:21.625225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.641065+0000","last_change":"2023-11-24T06:20:21.639872+0000","last_active":"2023-11-24T06:20:21.641065+0000","last_peered":"2023-11-24T06:20:21.641065+0000","last_clean":"2023-11-24T06:20:21.641065+0000","last_became_active":"2023-11-24T06:20:21.639600+0000","last_became_peered":"2023-11-24T06:20:21.639600+0000","last_unstale":"2023-11-24T06:20:21.641065+0000","last_undegraded":"2023-11-24T06:20:21.641065+0000","last_fullsized":"2023-11-24T06:20:21.641065+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:15.519175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.625436+0000","last_change":"2023-11-24T06:20:21.625436+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.625436+0000","last_undegraded":"2023-11-24T06:20:21.625436+0000","last_fullsized":"2023-11-24T06:20:21.625436+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639363+0000","last_change":"2023-11-24T06:20:21.638221+0000","last_active":"2023-11-24T06:20:21.639363+0000","last_peered":"2023-11-24T06:20:21.639363+0000","last_clean":"2023-11-24T06:20:21.639363+0000","last_became_active":"2023-11-24T06:20:21.637800+0000","last_became_peered":"2023-11-24T06:20:21.637800+0000","last_unstale":"2023-11-24T06:20:21.639363+0000","last_undegraded":"2023-11-24T06:20:21.639363+0000","last_fullsized":"2023-11-24T06:20:21.639363+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:23.447147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.628285+0000","last_change":"2023-11-24T06:20:21.628285+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.628285+0000","last_undegraded":"2023-11-24T06:20:21.628285+0000","last_fullsized":"2023-11-24T06:20:21.628285+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640027+0000","last_change":"2023-11-24T06:20:21.635897+0000","last_active":"2023-11-24T06:20:21.640027+0000","last_peered":"2023-11-24T06:20:21.640027+0000","last_clean":"2023-11-24T06:20:21.640027+0000","last_became_active":"2023-11-24T06:20:21.635167+0000","last_became_peered":"2023-11-24T06:20:21.635167+0000","last_unstale":"2023-11-24T06:20:21.640027+0000","last_undegraded":"2023-11-24T06:20:21.640027+0000","last_fullsized":"2023-11-24T06:20:21.640027+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:54:05.115719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626755+0000","last_change":"2023-11-24T06:20:21.626755+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626755+0000","last_undegraded":"2023-11-24T06:20:21.626755+0000","last_fullsized":"2023-11-24T06:20:21.626755+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638367+0000","last_change":"2023-11-24T06:20:21.633952+0000","last_active":"2023-11-24T06:20:21.638367+0000","last_peered":"2023-11-24T06:20:21.638367+0000","last_clean":"2023-11-24T06:20:21.638367+0000","last_became_active":"2023-11-24T06:20:21.633629+0000","last_became_peered":"2023-11-24T06:20:21.633629+0000","last_unstale":"2023-11-24T06:20:21.638367+0000","last_undegraded":"2023-11-24T06:20:21.638367+0000","last_fullsized":"2023-11-24T06:20:21.638367+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:53.363443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626539+0000","last_change":"2023-11-24T06:20:21.626539+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626539+0000","last_undegraded":"2023-11-24T06:20:21.626539+0000","last_fullsized":"2023-11-24T06:20:21.626539+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638377+0000","last_change":"2023-11-24T06:20:21.634883+0000","last_active":"2023-11-24T06:20:21.638377+0000","last_peered":"2023-11-24T06:20:21.638377+0000","last_clean":"2023-11-24T06:20:21.638377+0000","last_became_active":"2023-11-24T06:20:21.634624+0000","last_became_peered":"2023-11-24T06:20:21.634624+0000","last_unstale":"2023-11-24T06:20:21.638377+0000","last_undegraded":"2023-11-24T06:20:21.638377+0000","last_fullsized":"2023-11-24T06:20:21.638377+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:38.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622724+0000","last_change":"2023-11-24T06:20:21.622724+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622724+0000","last_undegraded":"2023-11-24T06:20:21.622724+0000","last_fullsized":"2023-11-24T06:20:21.622724+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.637356+0000","last_change":"2023-11-24T06:20:21.635390+0000","last_active":"2023-11-24T06:20:21.637356+0000","last_peered":"2023-11-24T06:20:21.637356+0000","last_clean":"2023-11-24T06:20:21.637356+0000","last_became_active":"2023-11-24T06:20:21.635114+0000","last_became_peered":"2023-11-24T06:20:21.635114+0000","last_unstale":"2023-11-24T06:20:21.637356+0000","last_undegraded":"2023-11-24T06:20:21.637356+0000","last_fullsized":"2023-11-24T06:20:21.637356+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:15:39.025211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.621069+0000","last_change":"2023-11-24T06:20:21.621069+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.621069+0000","last_undegraded":"2023-11-24T06:20:21.621069+0000","last_fullsized":"2023-11-24T06:20:21.621069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.640811+0000","last_change":"2023-11-24T06:20:21.636357+0000","last_active":"2023-11-24T06:20:21.640811+0000","last_peered":"2023-11-24T06:20:21.640811+0000","last_clean":"2023-11-24T06:20:21.640811+0000","last_became_active":"2023-11-24T06:20:21.636068+0000","last_became_peered":"2023-11-24T06:20:21.636068+0000","last_unstale":"2023-11-24T06:20:21.640811+0000","last_undegraded":"2023-11-24T06:20:21.640811+0000","last_fullsized":"2023-11-24T06:20:21.640811+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:24.432980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.626644+0000","last_change":"2023-11-24T06:20:21.626644+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.626644+0000","last_undegraded":"2023-11-24T06:20:21.626644+0000","last_fullsized":"2023-11-24T06:20:21.626644+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.641635+0000","last_change":"2023-11-24T06:20:21.641031+0000","last_active":"2023-11-24T06:20:21.641635+0000","last_peered":"2023-11-24T06:20:21.641635+0000","last_clean":"2023-11-24T06:20:21.641635+0000","last_became_active":"2023-11-24T06:20:21.640826+0000","last_became_peered":"2023-11-24T06:20:21.640826+0000","last_unstale":"2023-11-24T06:20:21.641635+0000","last_undegraded":"2023-11-24T06:20:21.641635+0000","last_fullsized":"2023-11-24T06:20:21.641635+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:20.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.622735+0000","last_change":"2023-11-24T06:20:21.622735+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.622735+0000","last_undegraded":"2023-11-24T06:20:21.622735+0000","last_fullsized":"2023-11-24T06:20:21.622735+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.639612+0000","last_change":"2023-11-24T06:20:21.636646+0000","last_active":"2023-11-24T06:20:21.639612+0000","last_peered":"2023-11-24T06:20:21.639612+0000","last_clean":"2023-11-24T06:20:21.639612+0000","last_became_active":"2023-11-24T06:20:21.636335+0000","last_became_peered":"2023-11-24T06:20:21.636335+0000","last_unstale":"2023-11-24T06:20:21.639612+0000","last_undegraded":"2023-11-24T06:20:21.639612+0000","last_fullsized":"2023-11-24T06:20:21.639612+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":51,"state":"creating+peering","last_fresh":"2023-11-24T06:20:21.623457+0000","last_change":"2023-11-24T06:20:21.623457+0000","last_active":"2023-11-24T06:20:21.614935+0000","last_peered":"2023-11-24T06:20:21.614935+0000","last_clean":"2023-11-24T06:20:21.614935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:21.623457+0000","last_undegraded":"2023-11-24T06:20:21.623457+0000","last_fullsized":"2023-11-24T06:20:21.623457+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-24T06:20:21.638494+0000","last_change":"2023-11-24T06:20:21.635744+0000","last_active":"2023-11-24T06:20:21.638494+0000","last_peered":"2023-11-24T06:20:21.638494+0000","last_clean":"2023-11-24T06:20:21.638494+0000","last_became_active":"2023-11-24T06:20:21.635447+0000","last_became_peered":"2023-11-24T06:20:21.635447+0000","last_unstale":"2023-11-24T06:20:21.638494+0000","last_undegraded":"2023-11-24T06:20:21.638494+0000","last_fullsized":"2023-11-24T06:20:21.638494+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:16:17.983184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-11-24T06:20:20.618880+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:20.618880+0000","last_peered":"2023-11-24T06:20:20.618880+0000","last_clean":"2023-11-24T06:20:20.618880+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:20.618880+0000","last_undegraded":"2023-11-24T06:20:20.618880+0000","last_fullsized":"2023-11-24T06:20:20.618880+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":841444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.70899999999999996}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.81000000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.78000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":6.423,"5min":6.423,"15min":6.423},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":12.554,"5min":12.554,"15min":12.554},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.75700000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.81599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":257343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":841444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.255}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.30599999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.373}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.61199999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.56200000000000006}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.57299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.41799999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.37},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.58799999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.89200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.223}]},{"osd":1,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.246},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.28899999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.317},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.30199999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.434},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53500000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.627},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67500000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.66200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":251076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.04,"5min":3.04,"15min":3.04},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":10.198,"5min":10.198,"15min":10.198},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":10.215999999999999,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.63500000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.66000000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:20:27.517 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3f is not active+clean 2023-11-24T06:20:27.518 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623555+0000', 'last_change': '2023-11-24T06:20:21.623555+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623555+0000', 'last_undegraded': '2023-11-24T06:20:21.623555+0000', 'last_fullsized': '2023-11-24T06:20:21.623555+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:20:27.518 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3e is not active+clean 2023-11-24T06:20:27.518 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.628738+0000', 'last_change': '2023-11-24T06:20:21.628738+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.628738+0000', 'last_undegraded': '2023-11-24T06:20:21.628738+0000', 'last_fullsized': '2023-11-24T06:20:21.628738+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.518 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3d is not active+clean 2023-11-24T06:20:27.518 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623537+0000', 'last_change': '2023-11-24T06:20:21.623537+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623537+0000', 'last_undegraded': '2023-11-24T06:20:21.623537+0000', 'last_fullsized': '2023-11-24T06:20:21.623537+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.518 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2023-11-24T06:20:27.518 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.626348+0000', 'last_change': '2023-11-24T06:20:21.626348+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.626348+0000', 'last_undegraded': '2023-11-24T06:20:21.626348+0000', 'last_fullsized': '2023-11-24T06:20:21.626348+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:20:27.518 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3b is not active+clean 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.624458+0000', 'last_change': '2023-11-24T06:20:21.624458+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.624458+0000', 'last_undegraded': '2023-11-24T06:20:21.624458+0000', 'last_fullsized': '2023-11-24T06:20:21.624458+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.627324+0000', 'last_change': '2023-11-24T06:20:21.627324+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.627324+0000', 'last_undegraded': '2023-11-24T06:20:21.627324+0000', 'last_fullsized': '2023-11-24T06:20:21.627324+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.39 is not active+clean 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.622653+0000', 'last_change': '2023-11-24T06:20:21.622653+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.622653+0000', 'last_undegraded': '2023-11-24T06:20:21.622653+0000', 'last_fullsized': '2023-11-24T06:20:21.622653+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.38 is not active+clean 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.629361+0000', 'last_change': '2023-11-24T06:20:21.629361+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.629361+0000', 'last_undegraded': '2023-11-24T06:20:21.629361+0000', 'last_fullsized': '2023-11-24T06:20:21.629361+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.37 is not active+clean 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.620894+0000', 'last_change': '2023-11-24T06:20:21.620894+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.620894+0000', 'last_undegraded': '2023-11-24T06:20:21.620894+0000', 'last_fullsized': '2023-11-24T06:20:21.620894+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:20:27.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.36 is not active+clean 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.626826+0000', 'last_change': '2023-11-24T06:20:21.626826+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.626826+0000', 'last_undegraded': '2023-11-24T06:20:21.626826+0000', 'last_fullsized': '2023-11-24T06:20:21.626826+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.35 is not active+clean 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.630536+0000', 'last_change': '2023-11-24T06:20:21.630536+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.630536+0000', 'last_undegraded': '2023-11-24T06:20:21.630536+0000', 'last_fullsized': '2023-11-24T06:20:21.630536+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625573+0000', 'last_change': '2023-11-24T06:20:21.625573+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625573+0000', 'last_undegraded': '2023-11-24T06:20:21.625573+0000', 'last_fullsized': '2023-11-24T06:20:21.625573+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.33 is not active+clean 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.626066+0000', 'last_change': '2023-11-24T06:20:21.626066+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.626066+0000', 'last_undegraded': '2023-11-24T06:20:21.626066+0000', 'last_fullsized': '2023-11-24T06:20:21.626066+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.32 is not active+clean 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.622517+0000', 'last_change': '2023-11-24T06:20:21.622517+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.622517+0000', 'last_undegraded': '2023-11-24T06:20:21.622517+0000', 'last_fullsized': '2023-11-24T06:20:21.622517+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:20:27.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.31 is not active+clean 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.620882+0000', 'last_change': '2023-11-24T06:20:21.620882+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.620882+0000', 'last_undegraded': '2023-11-24T06:20:21.620882+0000', 'last_fullsized': '2023-11-24T06:20:21.620882+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.624061+0000', 'last_change': '2023-11-24T06:20:21.624061+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.624061+0000', 'last_undegraded': '2023-11-24T06:20:21.624061+0000', 'last_fullsized': '2023-11-24T06:20:21.624061+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2f is not active+clean 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.626805+0000', 'last_change': '2023-11-24T06:20:21.626805+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.626805+0000', 'last_undegraded': '2023-11-24T06:20:21.626805+0000', 'last_fullsized': '2023-11-24T06:20:21.626805+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2e is not active+clean 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623770+0000', 'last_change': '2023-11-24T06:20:21.623770+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623770+0000', 'last_undegraded': '2023-11-24T06:20:21.623770+0000', 'last_fullsized': '2023-11-24T06:20:21.623770+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2d is not active+clean 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.624008+0000', 'last_change': '2023-11-24T06:20:21.624008+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.624008+0000', 'last_undegraded': '2023-11-24T06:20:21.624008+0000', 'last_fullsized': '2023-11-24T06:20:21.624008+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:20:27.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2c is not active+clean 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625942+0000', 'last_change': '2023-11-24T06:20:21.625942+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625942+0000', 'last_undegraded': '2023-11-24T06:20:21.625942+0000', 'last_fullsized': '2023-11-24T06:20:21.625942+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2b is not active+clean 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623073+0000', 'last_change': '2023-11-24T06:20:21.623073+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623073+0000', 'last_undegraded': '2023-11-24T06:20:21.623073+0000', 'last_fullsized': '2023-11-24T06:20:21.623073+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2a is not active+clean 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625109+0000', 'last_change': '2023-11-24T06:20:21.625109+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625109+0000', 'last_undegraded': '2023-11-24T06:20:21.625109+0000', 'last_fullsized': '2023-11-24T06:20:21.625109+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.29 is not active+clean 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.622512+0000', 'last_change': '2023-11-24T06:20:21.622512+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.622512+0000', 'last_undegraded': '2023-11-24T06:20:21.622512+0000', 'last_fullsized': '2023-11-24T06:20:21.622512+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.28 is not active+clean 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.627183+0000', 'last_change': '2023-11-24T06:20:21.627183+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.627183+0000', 'last_undegraded': '2023-11-24T06:20:21.627183+0000', 'last_fullsized': '2023-11-24T06:20:21.627183+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:20:27.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.27 is not active+clean 2023-11-24T06:20:27.523 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623106+0000', 'last_change': '2023-11-24T06:20:21.623106+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623106+0000', 'last_undegraded': '2023-11-24T06:20:21.623106+0000', 'last_fullsized': '2023-11-24T06:20:21.623106+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:20:27.523 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2023-11-24T06:20:27.523 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625894+0000', 'last_change': '2023-11-24T06:20:21.625894+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625894+0000', 'last_undegraded': '2023-11-24T06:20:21.625894+0000', 'last_fullsized': '2023-11-24T06:20:21.625894+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:20:27.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2023-11-24T06:20:27.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.627857+0000', 'last_change': '2023-11-24T06:20:21.627857+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.627857+0000', 'last_undegraded': '2023-11-24T06:20:21.627857+0000', 'last_fullsized': '2023-11-24T06:20:21.627857+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.24 is not active+clean 2023-11-24T06:20:27.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.622254+0000', 'last_change': '2023-11-24T06:20:21.622254+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.622254+0000', 'last_undegraded': '2023-11-24T06:20:21.622254+0000', 'last_fullsized': '2023-11-24T06:20:21.622254+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:20:27.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.23 is not active+clean 2023-11-24T06:20:27.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625015+0000', 'last_change': '2023-11-24T06:20:21.625015+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625015+0000', 'last_undegraded': '2023-11-24T06:20:21.625015+0000', 'last_fullsized': '2023-11-24T06:20:21.625015+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.22 is not active+clean 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621840+0000', 'last_change': '2023-11-24T06:20:21.621840+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621840+0000', 'last_undegraded': '2023-11-24T06:20:21.621840+0000', 'last_fullsized': '2023-11-24T06:20:21.621840+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.21 is not active+clean 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621760+0000', 'last_change': '2023-11-24T06:20:21.621760+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621760+0000', 'last_undegraded': '2023-11-24T06:20:21.621760+0000', 'last_fullsized': '2023-11-24T06:20:21.621760+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.624542+0000', 'last_change': '2023-11-24T06:20:21.624542+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.624542+0000', 'last_undegraded': '2023-11-24T06:20:21.624542+0000', 'last_fullsized': '2023-11-24T06:20:21.624542+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.626103+0000', 'last_change': '2023-11-24T06:20:21.626103+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.626103+0000', 'last_undegraded': '2023-11-24T06:20:21.626103+0000', 'last_fullsized': '2023-11-24T06:20:21.626103+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1f is not active+clean 2023-11-24T06:20:27.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.631414+0000', 'last_change': '2023-11-24T06:20:21.631414+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.631414+0000', 'last_undegraded': '2023-11-24T06:20:21.631414+0000', 'last_fullsized': '2023-11-24T06:20:21.631414+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625721+0000', 'last_change': '2023-11-24T06:20:21.625721+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625721+0000', 'last_undegraded': '2023-11-24T06:20:21.625721+0000', 'last_fullsized': '2023-11-24T06:20:21.625721+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.627331+0000', 'last_change': '2023-11-24T06:20:21.627331+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.627331+0000', 'last_undegraded': '2023-11-24T06:20:21.627331+0000', 'last_fullsized': '2023-11-24T06:20:21.627331+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623259+0000', 'last_change': '2023-11-24T06:20:21.623259+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623259+0000', 'last_undegraded': '2023-11-24T06:20:21.623259+0000', 'last_fullsized': '2023-11-24T06:20:21.623259+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623607+0000', 'last_change': '2023-11-24T06:20:21.623607+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623607+0000', 'last_undegraded': '2023-11-24T06:20:21.623607+0000', 'last_fullsized': '2023-11-24T06:20:21.623607+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625448+0000', 'last_change': '2023-11-24T06:20:21.625448+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625448+0000', 'last_undegraded': '2023-11-24T06:20:21.625448+0000', 'last_fullsized': '2023-11-24T06:20:21.625448+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.624564+0000', 'last_change': '2023-11-24T06:20:21.624564+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.624564+0000', 'last_undegraded': '2023-11-24T06:20:21.624564+0000', 'last_fullsized': '2023-11-24T06:20:21.624564+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621408+0000', 'last_change': '2023-11-24T06:20:21.621408+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621408+0000', 'last_undegraded': '2023-11-24T06:20:21.621408+0000', 'last_fullsized': '2023-11-24T06:20:21.621408+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621969+0000', 'last_change': '2023-11-24T06:20:21.621969+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621969+0000', 'last_undegraded': '2023-11-24T06:20:21.621969+0000', 'last_fullsized': '2023-11-24T06:20:21.621969+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.14 is not active+clean 2023-11-24T06:20:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.627272+0000', 'last_change': '2023-11-24T06:20:21.627272+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.627272+0000', 'last_undegraded': '2023-11-24T06:20:21.627272+0000', 'last_fullsized': '2023-11-24T06:20:21.627272+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621600+0000', 'last_change': '2023-11-24T06:20:21.621600+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621600+0000', 'last_undegraded': '2023-11-24T06:20:21.621600+0000', 'last_fullsized': '2023-11-24T06:20:21.621600+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.627692+0000', 'last_change': '2023-11-24T06:20:21.627692+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.627692+0000', 'last_undegraded': '2023-11-24T06:20:21.627692+0000', 'last_fullsized': '2023-11-24T06:20:21.627692+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.622661+0000', 'last_change': '2023-11-24T06:20:21.622661+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.622661+0000', 'last_undegraded': '2023-11-24T06:20:21.622661+0000', 'last_fullsized': '2023-11-24T06:20:21.622661+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621256+0000', 'last_change': '2023-11-24T06:20:21.621256+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621256+0000', 'last_undegraded': '2023-11-24T06:20:21.621256+0000', 'last_fullsized': '2023-11-24T06:20:21.621256+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2023-11-24T06:20:27.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621164+0000', 'last_change': '2023-11-24T06:20:21.621164+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621164+0000', 'last_undegraded': '2023-11-24T06:20:21.621164+0000', 'last_fullsized': '2023-11-24T06:20:21.621164+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621410+0000', 'last_change': '2023-11-24T06:20:21.621410+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621410+0000', 'last_undegraded': '2023-11-24T06:20:21.621410+0000', 'last_fullsized': '2023-11-24T06:20:21.621410+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.627180+0000', 'last_change': '2023-11-24T06:20:21.627180+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.627180+0000', 'last_undegraded': '2023-11-24T06:20:21.627180+0000', 'last_fullsized': '2023-11-24T06:20:21.627180+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623054+0000', 'last_change': '2023-11-24T06:20:21.623054+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623054+0000', 'last_undegraded': '2023-11-24T06:20:21.623054+0000', 'last_fullsized': '2023-11-24T06:20:21.623054+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621052+0000', 'last_change': '2023-11-24T06:20:21.621052+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621052+0000', 'last_undegraded': '2023-11-24T06:20:21.621052+0000', 'last_fullsized': '2023-11-24T06:20:21.621052+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2023-11-24T06:20:27.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625137+0000', 'last_change': '2023-11-24T06:20:21.625137+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625137+0000', 'last_undegraded': '2023-11-24T06:20:21.625137+0000', 'last_fullsized': '2023-11-24T06:20:21.625137+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:20:27.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2023-11-24T06:20:27.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625641+0000', 'last_change': '2023-11-24T06:20:21.625641+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625641+0000', 'last_undegraded': '2023-11-24T06:20:21.625641+0000', 'last_fullsized': '2023-11-24T06:20:21.625641+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2023-11-24T06:20:27.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625225+0000', 'last_change': '2023-11-24T06:20:21.625225+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625225+0000', 'last_undegraded': '2023-11-24T06:20:21.625225+0000', 'last_fullsized': '2023-11-24T06:20:21.625225+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2023-11-24T06:20:27.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.625436+0000', 'last_change': '2023-11-24T06:20:21.625436+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.625436+0000', 'last_undegraded': '2023-11-24T06:20:21.625436+0000', 'last_fullsized': '2023-11-24T06:20:21.625436+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:20:27.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2023-11-24T06:20:27.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.628285+0000', 'last_change': '2023-11-24T06:20:21.628285+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.628285+0000', 'last_undegraded': '2023-11-24T06:20:21.628285+0000', 'last_fullsized': '2023-11-24T06:20:21.628285+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.4 is not active+clean 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.626755+0000', 'last_change': '2023-11-24T06:20:21.626755+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.626755+0000', 'last_undegraded': '2023-11-24T06:20:21.626755+0000', 'last_fullsized': '2023-11-24T06:20:21.626755+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.626539+0000', 'last_change': '2023-11-24T06:20:21.626539+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.626539+0000', 'last_undegraded': '2023-11-24T06:20:21.626539+0000', 'last_fullsized': '2023-11-24T06:20:21.626539+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.622724+0000', 'last_change': '2023-11-24T06:20:21.622724+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.622724+0000', 'last_undegraded': '2023-11-24T06:20:21.622724+0000', 'last_fullsized': '2023-11-24T06:20:21.622724+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.621069+0000', 'last_change': '2023-11-24T06:20:21.621069+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.621069+0000', 'last_undegraded': '2023-11-24T06:20:21.621069+0000', 'last_fullsized': '2023-11-24T06:20:21.621069+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:20:27.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.0 is not active+clean 2023-11-24T06:20:27.533 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.626644+0000', 'last_change': '2023-11-24T06:20:21.626644+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.626644+0000', 'last_undegraded': '2023-11-24T06:20:21.626644+0000', 'last_fullsized': '2023-11-24T06:20:21.626644+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:20:27.533 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2023-11-24T06:20:27.533 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.622735+0000', 'last_change': '2023-11-24T06:20:21.622735+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.622735+0000', 'last_undegraded': '2023-11-24T06:20:21.622735+0000', 'last_fullsized': '2023-11-24T06:20:21.622735+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.533 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2023-11-24T06:20:27.533 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 51, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:21.623457+0000', 'last_change': '2023-11-24T06:20:21.623457+0000', 'last_active': '2023-11-24T06:20:21.614935+0000', 'last_peered': '2023-11-24T06:20:21.614935+0000', 'last_clean': '2023-11-24T06:20:21.614935+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:21.623457+0000', 'last_undegraded': '2023-11-24T06:20:21.623457+0000', 'last_fullsized': '2023-11-24T06:20:21.623457+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:21.614935+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:20:27.534 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-11-24T06:20:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.671+0000 7f2beaacf700 1 Processor -- start 2023-11-24T06:20:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.671+0000 7f2beaacf700 1 -- start start 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2beaacf700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2be415e080 0x7f2be40b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2beaacf700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40ae430 con 0x7f2be406c710 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2beaacf700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40ae710 con 0x7f2be40af4d0 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2beaacf700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40af0b0 con 0x7f2be415e080 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be906c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2be406c710 0x7f2be406e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41516/0 (socket says 172.21.15.89:41516) 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be906c700 1 -- 172.21.15.89:0/4103003636 learned_addr learned my addr 172.21.15.89:0/4103003636 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621543764 0 0) 0x7f2be40ae430 con 0x7f2be406c710 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd40035b0 con 0x7f2be406c710 2023-11-24T06:20:27.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be886b700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2be415e080 0x7f2be40b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2192771961 0 0) 0x7f2bd40035b0 con 0x7f2be406c710 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2be415e080 msgr2=0x7f2be40b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2be415e080 0x7f2be40b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 >> v1:172.21.15.123:6789/0 conn(0x7f2be40af4d0 legacy=0x7f2be406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40ae9f0 con 0x7f2be406c710 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f2bdc0019f0 con 0x7f2be406c710 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 >> v1:172.21.15.89:6789/0 conn(0x7f2be406c710 legacy=0x7f2be406e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bd4003a30 0x7f2bd4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4004850 0x7f2bd4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.672+0000 7f2be37fe700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2be415e080 0x7f2bd4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40af0b0 con 0x7f2bd4004850 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd400bac0 con 0x7f2bd4003a30 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd400bc80 con 0x7f2be415e080 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be3fff700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4004850 0x7f2bd4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be886b700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bd4003a30 0x7f2bd4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be906c700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2be415e080 0x7f2bd4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be3fff700 1 -- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2be415e080 msgr2=0x7f2bd4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be3fff700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2be415e080 0x7f2bd4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be3fff700 1 -- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bd4003a30 msgr2=0x7f2bd4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be3fff700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bd4003a30 0x7f2bd4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be3fff700 1 -- 172.21.15.89:0/4103003636 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd40035b0 con 0x7f2bd4004850 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be886b700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bd4003a30 0x7f2bd4003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be3fff700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4004850 0x7f2bd4004cd0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0007fd0 tx=0x7f2bd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1625ab2cad7bda61 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd00052d0 con 0x7f2bd4004850 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd0005430 con 0x7f2bd4004850 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.673+0000 7f2be37fe700 1 -- 172.21.15.89:0/4103003636 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd00038c0 con 0x7f2bd4004850 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.674+0000 7f2beaacf700 1 -- 172.21.15.89:0/4103003636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4004850 msgr2=0x7f2bd4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.674+0000 7f2beaacf700 1 --2- 172.21.15.89:0/4103003636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4004850 0x7f2bd4004cd0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0007fd0 tx=0x7f2bd4005880 comp rx=0 tx=0).stop 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.674+0000 7f2be906c700 1 -- 172.21.15.89:0/4103003636 reap_dead start 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.674+0000 7f2beaacf700 1 -- 172.21.15.89:0/4103003636 shutdown_connections 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.674+0000 7f2beaacf700 1 -- 172.21.15.89:0/4103003636 >> 172.21.15.89:0/4103003636 conn(0x7f2be4071b90 msgr2=0x7f2be4071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.674+0000 7f2beaacf700 1 -- 172.21.15.89:0/4103003636 shutdown_connections 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.674+0000 7f2beaacf700 1 -- 172.21.15.89:0/4103003636 wait complete. 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.675+0000 7f2beaacf700 1 Processor -- start 2023-11-24T06:20:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.675+0000 7f2beaacf700 1 -- start start 2023-11-24T06:20:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2beaacf700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2bd4004850 0x7f2be4146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2beaacf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4157b40 0x7f2be4155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2beaacf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be4152860 0x7f2be4155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2beaacf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be4146ab0 con 0x7f2be4152860 2023-11-24T06:20:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2beaacf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be415e460 con 0x7f2be4157b40 2023-11-24T06:20:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2beaacf700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be41580b0 con 0x7f2bd4004850 2023-11-24T06:20:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be4152860 0x7f2be4155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be4152860 0x7f2be4155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38024/0 (socket says 172.21.15.89:38024) 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 -- 172.21.15.89:0/3533692583 learned_addr learned my addr 172.21.15.89:0/3533692583 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be3fff700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4157b40 0x7f2be4155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be886b700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2bd4004850 0x7f2be4146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 -- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2bd4004850 msgr2=0x7f2be4146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2bd4004850 0x7f2be4146570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 -- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4157b40 msgr2=0x7f2be4155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4157b40 0x7f2be4155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 -- 172.21.15.89:0/3533692583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be4147f60 con 0x7f2be4152860 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be3fff700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4157b40 0x7f2be4155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.676+0000 7f2be906c700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be4152860 0x7f2be4155770 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2be4072a90 tx=0x7f2bdc00a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.677+0000 7f2be17fa700 1 -- 172.21.15.89:0/3533692583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc00ae20 con 0x7f2be4152860 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.677+0000 7f2be17fa700 1 -- 172.21.15.89:0/3533692583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc0133f0 con 0x7f2be4152860 2023-11-24T06:20:27.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.677+0000 7f2be17fa700 1 -- 172.21.15.89:0/3533692583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc0137a0 con 0x7f2be4152860 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.677+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be4148240 con 0x7f2be4152860 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.677+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be4152f40 con 0x7f2be4152860 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.677+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be40753f0 con 0x7f2be4152860 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.678+0000 7f2be17fa700 1 -- 172.21.15.89:0/3533692583 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2bdc013900 con 0x7f2be4152860 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.678+0000 7f2be17fa700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2bc806aed0 0x7f2bc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.679+0000 7f2be17fa700 1 -- 172.21.15.89:0/3533692583 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bdc08a060 con 0x7f2be4152860 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.679+0000 7f2be886b700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2bc806aed0 0x7f2bc806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.679+0000 7f2be886b700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2bc806aed0 0x7f2bc806d390 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009b20 tx=0x7f2bd8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.683+0000 7f2be17fa700 1 -- 172.21.15.89:0/3533692583 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bdc055260 con 0x7f2be4152860 2023-11-24T06:20:27.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.955+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2be406ded0 con 0x7f2be4152860 2023-11-24T06:20:27.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.955+0000 7f2be17fa700 1 -- 172.21.15.89:0/3533692583 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1452 (secure 0 0 0) 0x7f2bdc0175c0 con 0x7f2be4152860 2023-11-24T06:20:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:27.959 INFO:teuthology.orchestra.run.smithi089.stdout:{"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":123,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":53,"num_osds":8,"num_up_osds":8,"osd_up_since":1700806715,"num_in_osds":8,"osd_in_since":1700806706,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":227352576,"bytes_avail":772866760704,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":31,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":5577}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-11-24T06:19:59.734084+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2bc806aed0 msgr2=0x7f2bc806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2bc806aed0 0x7f2bc806d390 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009b20 tx=0x7f2bd8008040 comp rx=0 tx=0).stop 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be4152860 msgr2=0x7f2be4155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be4152860 0x7f2be4155770 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2be4072a90 tx=0x7f2bdc00a5f0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 shutdown_connections 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be4152860 0x7f2be4155770 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2bd4004850 0x7f2be4146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2bc806aed0 0x7f2bc806d390 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 --2- 172.21.15.89:0/3533692583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4157b40 0x7f2be4155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 >> 172.21.15.89:0/3533692583 conn(0x7f2be4071b90 msgr2=0x7f2be40b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 shutdown_connections 2023-11-24T06:20:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:27.958+0000 7f2beaacf700 1 -- 172.21.15.89:0/3533692583 wait complete. 2023-11-24T06:20:27.973 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-11-24T06:20:27.973 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:20:28.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.090+0000 7fb4fff55700 1 Processor -- start 2023-11-24T06:20:28.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.090+0000 7fb4fff55700 1 -- start start 2023-11-24T06:20:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fff55700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f80b49a0 0x7fb4f80d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fff55700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f815c4b0 con 0x7fb4f8152cc0 2023-11-24T06:20:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fff55700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f815c790 con 0x7fb4f806c710 2023-11-24T06:20:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fff55700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815ca70 con 0x7fb4f80b49a0 2023-11-24T06:20:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fdcf1700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb4f8152cc0 0x7fb4f806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41532/0 (socket says 172.21.15.89:41532) 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fdcf1700 1 -- 172.21.15.89:0/1488003997 learned_addr learned my addr 172.21.15.89:0/1488003997 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522028733 0 0) 0x7fb4f815c4b0 con 0x7fb4f8152cc0 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e40035b0 con 0x7fb4f8152cc0 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fe4f2700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f80b49a0 0x7fb4f80d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899696379 0 0) 0x7fb4e40035b0 con 0x7fb4f8152cc0 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f80b49a0 msgr2=0x7fb4f80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f80b49a0 0x7fb4f80d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 >> v1:172.21.15.123:6789/0 conn(0x7fb4f806c710 legacy=0x7fb4f80cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f815cd50 con 0x7fb4f8152cc0 2023-11-24T06:20:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fb4f4001670 con 0x7fb4f8152cc0 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.091+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 >> v1:172.21.15.89:6789/0 conn(0x7fb4f8152cc0 legacy=0x7fb4f806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e4003a30 0x7fb4e4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f80b49a0 0x7fb4e4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815ca70 con 0x7fb4e4004850 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e400bb70 con 0x7fb4e4003a30 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e400bd30 con 0x7fb4f80b49a0 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fd4f0700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fdcf1700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e4003a30 0x7fb4e4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fe4f2700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f80b49a0 0x7fb4e4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fd4f0700 1 -- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f80b49a0 msgr2=0x7fb4e4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:28.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fd4f0700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f80b49a0 0x7fb4e4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fd4f0700 1 -- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e4003a30 msgr2=0x7fb4e4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fd4f0700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e4003a30 0x7fb4e4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fd4f0700 1 -- 172.21.15.89:0/1488003997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e40035b0 con 0x7fb4e4004850 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fd4f0700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4e4004cd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80017c0 tx=0x7fb4e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe0d20fe27b9d09 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80052d0 con 0x7fb4e4004850 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e8005430 con 0x7fb4e4004850 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.092+0000 7fb4fccef700 1 -- 172.21.15.89:0/1488003997 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80038c0 con 0x7fb4e4004850 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.093+0000 7fb4fff55700 1 -- 172.21.15.89:0/1488003997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 msgr2=0x7fb4e4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.093+0000 7fb4fff55700 1 --2- 172.21.15.89:0/1488003997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4e4004cd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80017c0 tx=0x7fb4e4005880 comp rx=0 tx=0).stop 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.093+0000 7fb4fe4f2700 1 -- 172.21.15.89:0/1488003997 reap_dead start 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.093+0000 7fb4fff55700 1 -- 172.21.15.89:0/1488003997 shutdown_connections 2023-11-24T06:20:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.093+0000 7fb4fff55700 1 -- 172.21.15.89:0/1488003997 >> 172.21.15.89:0/1488003997 conn(0x7fb4f8071b90 msgr2=0x7fb4f8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.093+0000 7fb4fff55700 1 -- 172.21.15.89:0/1488003997 shutdown_connections 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.093+0000 7fb4fff55700 1 -- 172.21.15.89:0/1488003997 wait complete. 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.094+0000 7fb4fff55700 1 Processor -- start 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.094+0000 7fb4fff55700 1 -- start start 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fff55700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4f81480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fff55700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4f8147680 0x7fb4f814e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fff55700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f8155570 0x7fb4f81559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fff55700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f8149400 con 0x7fb4e4004850 2023-11-24T06:20:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fff55700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f8155f30 con 0x7fb4f8147680 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fff55700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f8147b00 con 0x7fb4f8155570 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4f81480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4f81480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38040/0 (socket says 172.21.15.89:38040) 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 -- 172.21.15.89:0/1627391180 learned_addr learned my addr 172.21.15.89:0/1627391180 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fe4f2700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f8155570 0x7fb4f81559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fd4f0700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4f8147680 0x7fb4f814e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 -- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f8155570 msgr2=0x7fb4f81559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f8155570 0x7fb4f81559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 -- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4f8147680 msgr2=0x7fb4f814e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4f8147680 0x7fb4f814e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 -- 172.21.15.89:0/1627391180 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f814e9f0 con 0x7fb4e4004850 2023-11-24T06:20:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fe4f2700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f8155570 0x7fb4f81559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fd4f0700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4f8147680 0x7fb4f814e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fdcf1700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4f81480f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4007290 tx=0x7fb4f40075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4ee7fc700 1 -- 172.21.15.89:0/1627391180 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f400f660 con 0x7fb4e4004850 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4ee7fc700 1 -- 172.21.15.89:0/1627391180 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f400f7c0 con 0x7fb4e4004850 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f8148780 con 0x7fb4e4004850 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4ee7fc700 1 -- 172.21.15.89:0/1627391180 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f400fb70 con 0x7fb4e4004850 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.095+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f8148d40 con 0x7fb4e4004850 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.096+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f80753f0 con 0x7fb4e4004850 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.097+0000 7fb4ee7fc700 1 -- 172.21.15.89:0/1627391180 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb4f400fcd0 con 0x7fb4e4004850 2023-11-24T06:20:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.097+0000 7fb4ee7fc700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4dc06aed0 0x7fb4dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:28.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.097+0000 7fb4fd4f0700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4dc06aed0 0x7fb4dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:28.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.097+0000 7fb4ee7fc700 1 -- 172.21.15.89:0/1627391180 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4f40895d0 con 0x7fb4e4004850 2023-11-24T06:20:28.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.098+0000 7fb4fd4f0700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4dc06aed0 0x7fb4dc06d390 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80b0fe0 tx=0x7fb4e8009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:28.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.102+0000 7fb4ee7fc700 1 -- 172.21.15.89:0/1627391180 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4f4054680 con 0x7fb4e4004850 2023-11-24T06:20:28.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.288+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4f80b4160 con 0x7fb4dc06aed0 2023-11-24T06:20:28.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.296+0000 7fb4ee7fc700 1 -- 172.21.15.89:0/1627391180 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289502 (secure 0 0 0) 0x7fb4f80b4160 con 0x7fb4dc06aed0 2023-11-24T06:20:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4dc06aed0 msgr2=0x7fb4dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4dc06aed0 0x7fb4dc06d390 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80b0fe0 tx=0x7fb4e8009030 comp rx=0 tx=0).stop 2023-11-24T06:20:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 msgr2=0x7fb4f81480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4f81480f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4007290 tx=0x7fb4f40075a0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 shutdown_connections 2023-11-24T06:20:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb4f8155570 0x7fb4f81559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004850 0x7fb4f81480f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4dc06aed0 0x7fb4dc06d390 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 --2- 172.21.15.89:0/1627391180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4f8147680 0x7fb4f814e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 >> 172.21.15.89:0/1627391180 conn(0x7fb4f8071b90 msgr2=0x7fb4f8151560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 shutdown_connections 2023-11-24T06:20:28.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:28.299+0000 7fb4fff55700 1 -- 172.21.15.89:0/1627391180 wait complete. 2023-11-24T06:20:28.307 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:20:28.314 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2023-11-24T06:20:27.739177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222368,"kb_used_data":6184,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752352,"statfs":{"total":773094113280,"available":772866408448,"internally_reserved":0,"allocated":6332416,"data_stored":3348900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.121313"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949353+0000","last_change":"2023-11-24T06:20:22.653131+0000","last_active":"2023-11-24T06:20:23.949353+0000","last_peered":"2023-11-24T06:20:23.949353+0000","last_clean":"2023-11-24T06:20:23.949353+0000","last_became_active":"2023-11-24T06:20:22.652875+0000","last_became_peered":"2023-11-24T06:20:22.652875+0000","last_unstale":"2023-11-24T06:20:23.949353+0000","last_undegraded":"2023-11-24T06:20:23.949353+0000","last_fullsized":"2023-11-24T06:20:23.949353+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:28:41.054467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953004+0000","last_change":"2023-11-24T06:20:22.656133+0000","last_active":"2023-11-24T06:20:23.953004+0000","last_peered":"2023-11-24T06:20:23.953004+0000","last_clean":"2023-11-24T06:20:23.953004+0000","last_became_active":"2023-11-24T06:20:22.655721+0000","last_became_peered":"2023-11-24T06:20:22.655721+0000","last_unstale":"2023-11-24T06:20:23.953004+0000","last_undegraded":"2023-11-24T06:20:23.953004+0000","last_fullsized":"2023-11-24T06:20:23.953004+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:03.760844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653343+0000","last_change":"2023-11-24T06:20:22.642395+0000","last_active":"2023-11-24T06:20:23.653343+0000","last_peered":"2023-11-24T06:20:23.653343+0000","last_clean":"2023-11-24T06:20:23.653343+0000","last_became_active":"2023-11-24T06:20:22.642101+0000","last_became_peered":"2023-11-24T06:20:22.642101+0000","last_unstale":"2023-11-24T06:20:23.653343+0000","last_undegraded":"2023-11-24T06:20:23.653343+0000","last_fullsized":"2023-11-24T06:20:23.653343+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:05.174311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953108+0000","last_change":"2023-11-24T06:20:22.652050+0000","last_active":"2023-11-24T06:20:23.953108+0000","last_peered":"2023-11-24T06:20:23.953108+0000","last_clean":"2023-11-24T06:20:23.953108+0000","last_became_active":"2023-11-24T06:20:22.651804+0000","last_became_peered":"2023-11-24T06:20:22.651804+0000","last_unstale":"2023-11-24T06:20:23.953108+0000","last_undegraded":"2023-11-24T06:20:23.953108+0000","last_fullsized":"2023-11-24T06:20:23.953108+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:56:33.904287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949240+0000","last_change":"2023-11-24T06:20:22.651948+0000","last_active":"2023-11-24T06:20:23.949240+0000","last_peered":"2023-11-24T06:20:23.949240+0000","last_clean":"2023-11-24T06:20:23.949240+0000","last_became_active":"2023-11-24T06:20:22.651653+0000","last_became_peered":"2023-11-24T06:20:22.651653+0000","last_unstale":"2023-11-24T06:20:23.949240+0000","last_undegraded":"2023-11-24T06:20:23.949240+0000","last_fullsized":"2023-11-24T06:20:23.949240+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:06.512497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952159+0000","last_change":"2023-11-24T06:20:22.653183+0000","last_active":"2023-11-24T06:20:23.952159+0000","last_peered":"2023-11-24T06:20:23.952159+0000","last_clean":"2023-11-24T06:20:23.952159+0000","last_became_active":"2023-11-24T06:20:22.652787+0000","last_became_peered":"2023-11-24T06:20:22.652787+0000","last_unstale":"2023-11-24T06:20:23.952159+0000","last_undegraded":"2023-11-24T06:20:23.952159+0000","last_fullsized":"2023-11-24T06:20:23.952159+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:45.138650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652864+0000","last_change":"2023-11-24T06:20:22.643286+0000","last_active":"2023-11-24T06:20:23.652864+0000","last_peered":"2023-11-24T06:20:23.652864+0000","last_clean":"2023-11-24T06:20:23.652864+0000","last_became_active":"2023-11-24T06:20:22.642994+0000","last_became_peered":"2023-11-24T06:20:22.642994+0000","last_unstale":"2023-11-24T06:20:23.652864+0000","last_undegraded":"2023-11-24T06:20:23.652864+0000","last_fullsized":"2023-11-24T06:20:23.652864+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:02:32.900382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952340+0000","last_change":"2023-11-24T06:20:22.648090+0000","last_active":"2023-11-24T06:20:23.952340+0000","last_peered":"2023-11-24T06:20:23.952340+0000","last_clean":"2023-11-24T06:20:23.952340+0000","last_became_active":"2023-11-24T06:20:22.647796+0000","last_became_peered":"2023-11-24T06:20:22.647796+0000","last_unstale":"2023-11-24T06:20:23.952340+0000","last_undegraded":"2023-11-24T06:20:23.952340+0000","last_fullsized":"2023-11-24T06:20:23.952340+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:03.457321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653032+0000","last_change":"2023-11-24T06:20:22.651119+0000","last_active":"2023-11-24T06:20:23.653032+0000","last_peered":"2023-11-24T06:20:23.653032+0000","last_clean":"2023-11-24T06:20:23.653032+0000","last_became_active":"2023-11-24T06:20:22.650793+0000","last_became_peered":"2023-11-24T06:20:22.650793+0000","last_unstale":"2023-11-24T06:20:23.653032+0000","last_undegraded":"2023-11-24T06:20:23.653032+0000","last_fullsized":"2023-11-24T06:20:23.653032+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:31.398961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652093+0000","last_change":"2023-11-24T06:20:22.645989+0000","last_active":"2023-11-24T06:20:23.652093+0000","last_peered":"2023-11-24T06:20:23.652093+0000","last_clean":"2023-11-24T06:20:23.652093+0000","last_became_active":"2023-11-24T06:20:22.645725+0000","last_became_peered":"2023-11-24T06:20:22.645725+0000","last_unstale":"2023-11-24T06:20:23.652093+0000","last_undegraded":"2023-11-24T06:20:23.652093+0000","last_fullsized":"2023-11-24T06:20:23.652093+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:35:46.501400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652229+0000","last_change":"2023-11-24T06:20:22.654068+0000","last_active":"2023-11-24T06:20:23.652229+0000","last_peered":"2023-11-24T06:20:23.652229+0000","last_clean":"2023-11-24T06:20:23.652229+0000","last_became_active":"2023-11-24T06:20:22.653765+0000","last_became_peered":"2023-11-24T06:20:22.653765+0000","last_unstale":"2023-11-24T06:20:23.652229+0000","last_undegraded":"2023-11-24T06:20:23.652229+0000","last_fullsized":"2023-11-24T06:20:23.652229+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:04:43.643723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953347+0000","last_change":"2023-11-24T06:20:22.648555+0000","last_active":"2023-11-24T06:20:23.953347+0000","last_peered":"2023-11-24T06:20:23.953347+0000","last_clean":"2023-11-24T06:20:23.953347+0000","last_became_active":"2023-11-24T06:20:22.646184+0000","last_became_peered":"2023-11-24T06:20:22.646184+0000","last_unstale":"2023-11-24T06:20:23.953347+0000","last_undegraded":"2023-11-24T06:20:23.953347+0000","last_fullsized":"2023-11-24T06:20:23.953347+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:18.409593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953412+0000","last_change":"2023-11-24T06:20:22.649705+0000","last_active":"2023-11-24T06:20:23.953412+0000","last_peered":"2023-11-24T06:20:23.953412+0000","last_clean":"2023-11-24T06:20:23.953412+0000","last_became_active":"2023-11-24T06:20:22.649413+0000","last_became_peered":"2023-11-24T06:20:22.649413+0000","last_unstale":"2023-11-24T06:20:23.953412+0000","last_undegraded":"2023-11-24T06:20:23.953412+0000","last_fullsized":"2023-11-24T06:20:23.953412+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:24.451689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653311+0000","last_change":"2023-11-24T06:20:22.650989+0000","last_active":"2023-11-24T06:20:23.653311+0000","last_peered":"2023-11-24T06:20:23.653311+0000","last_clean":"2023-11-24T06:20:23.653311+0000","last_became_active":"2023-11-24T06:20:22.650747+0000","last_became_peered":"2023-11-24T06:20:22.650747+0000","last_unstale":"2023-11-24T06:20:23.653311+0000","last_undegraded":"2023-11-24T06:20:23.653311+0000","last_fullsized":"2023-11-24T06:20:23.653311+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:41:46.472796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652932+0000","last_change":"2023-11-24T06:20:22.641140+0000","last_active":"2023-11-24T06:20:23.652932+0000","last_peered":"2023-11-24T06:20:23.652932+0000","last_clean":"2023-11-24T06:20:23.652932+0000","last_became_active":"2023-11-24T06:20:22.640830+0000","last_became_peered":"2023-11-24T06:20:22.640830+0000","last_unstale":"2023-11-24T06:20:23.652932+0000","last_undegraded":"2023-11-24T06:20:23.652932+0000","last_fullsized":"2023-11-24T06:20:23.652932+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:04:43.643723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952163+0000","last_change":"2023-11-24T06:20:22.646028+0000","last_active":"2023-11-24T06:20:23.952163+0000","last_peered":"2023-11-24T06:20:23.952163+0000","last_clean":"2023-11-24T06:20:23.952163+0000","last_became_active":"2023-11-24T06:20:22.645701+0000","last_became_peered":"2023-11-24T06:20:22.645701+0000","last_unstale":"2023-11-24T06:20:23.952163+0000","last_undegraded":"2023-11-24T06:20:23.952163+0000","last_fullsized":"2023-11-24T06:20:23.952163+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:20.360413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953148+0000","last_change":"2023-11-24T06:20:22.651149+0000","last_active":"2023-11-24T06:20:23.953148+0000","last_peered":"2023-11-24T06:20:23.953148+0000","last_clean":"2023-11-24T06:20:23.953148+0000","last_became_active":"2023-11-24T06:20:22.650825+0000","last_became_peered":"2023-11-24T06:20:22.650825+0000","last_unstale":"2023-11-24T06:20:23.953148+0000","last_undegraded":"2023-11-24T06:20:23.953148+0000","last_fullsized":"2023-11-24T06:20:23.953148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:25.860603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.654044+0000","last_change":"2023-11-24T06:20:22.650034+0000","last_active":"2023-11-24T06:20:23.654044+0000","last_peered":"2023-11-24T06:20:23.654044+0000","last_clean":"2023-11-24T06:20:23.654044+0000","last_became_active":"2023-11-24T06:20:22.649540+0000","last_became_peered":"2023-11-24T06:20:22.649540+0000","last_unstale":"2023-11-24T06:20:23.654044+0000","last_undegraded":"2023-11-24T06:20:23.654044+0000","last_fullsized":"2023-11-24T06:20:23.654044+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:11.136219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949211+0000","last_change":"2023-11-24T06:20:22.647946+0000","last_active":"2023-11-24T06:20:23.949211+0000","last_peered":"2023-11-24T06:20:23.949211+0000","last_clean":"2023-11-24T06:20:23.949211+0000","last_became_active":"2023-11-24T06:20:22.647680+0000","last_became_peered":"2023-11-24T06:20:22.647680+0000","last_unstale":"2023-11-24T06:20:23.949211+0000","last_undegraded":"2023-11-24T06:20:23.949211+0000","last_fullsized":"2023-11-24T06:20:23.949211+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:11:37.991620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952928+0000","last_change":"2023-11-24T06:20:22.648917+0000","last_active":"2023-11-24T06:20:23.952928+0000","last_peered":"2023-11-24T06:20:23.952928+0000","last_clean":"2023-11-24T06:20:23.952928+0000","last_became_active":"2023-11-24T06:20:22.648656+0000","last_became_peered":"2023-11-24T06:20:22.648656+0000","last_unstale":"2023-11-24T06:20:23.952928+0000","last_undegraded":"2023-11-24T06:20:23.952928+0000","last_fullsized":"2023-11-24T06:20:23.952928+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:56:08.700328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653517+0000","last_change":"2023-11-24T06:20:22.645991+0000","last_active":"2023-11-24T06:20:23.653517+0000","last_peered":"2023-11-24T06:20:23.653517+0000","last_clean":"2023-11-24T06:20:23.653517+0000","last_became_active":"2023-11-24T06:20:22.645726+0000","last_became_peered":"2023-11-24T06:20:22.645726+0000","last_unstale":"2023-11-24T06:20:23.653517+0000","last_undegraded":"2023-11-24T06:20:23.653517+0000","last_fullsized":"2023-11-24T06:20:23.653517+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:28:18.701104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952528+0000","last_change":"2023-11-24T06:20:22.654435+0000","last_active":"2023-11-24T06:20:23.952528+0000","last_peered":"2023-11-24T06:20:23.952528+0000","last_clean":"2023-11-24T06:20:23.952528+0000","last_became_active":"2023-11-24T06:20:22.653999+0000","last_became_peered":"2023-11-24T06:20:22.653999+0000","last_unstale":"2023-11-24T06:20:23.952528+0000","last_undegraded":"2023-11-24T06:20:23.952528+0000","last_fullsized":"2023-11-24T06:20:23.952528+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:54.507295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653240+0000","last_change":"2023-11-24T06:20:22.652948+0000","last_active":"2023-11-24T06:20:23.653240+0000","last_peered":"2023-11-24T06:20:23.653240+0000","last_clean":"2023-11-24T06:20:23.653240+0000","last_became_active":"2023-11-24T06:20:22.652706+0000","last_became_peered":"2023-11-24T06:20:22.652706+0000","last_unstale":"2023-11-24T06:20:23.653240+0000","last_undegraded":"2023-11-24T06:20:23.653240+0000","last_fullsized":"2023-11-24T06:20:23.653240+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:18.409593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952649+0000","last_change":"2023-11-24T06:20:22.654236+0000","last_active":"2023-11-24T06:20:23.952649+0000","last_peered":"2023-11-24T06:20:23.952649+0000","last_clean":"2023-11-24T06:20:23.952649+0000","last_became_active":"2023-11-24T06:20:22.654019+0000","last_became_peered":"2023-11-24T06:20:22.654019+0000","last_unstale":"2023-11-24T06:20:23.952649+0000","last_undegraded":"2023-11-24T06:20:23.952649+0000","last_fullsized":"2023-11-24T06:20:23.952649+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:06.068523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949341+0000","last_change":"2023-11-24T06:20:22.651422+0000","last_active":"2023-11-24T06:20:23.949341+0000","last_peered":"2023-11-24T06:20:23.949341+0000","last_clean":"2023-11-24T06:20:23.949341+0000","last_became_active":"2023-11-24T06:20:22.651140+0000","last_became_peered":"2023-11-24T06:20:22.651140+0000","last_unstale":"2023-11-24T06:20:23.949341+0000","last_undegraded":"2023-11-24T06:20:23.949341+0000","last_fullsized":"2023-11-24T06:20:23.949341+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:57:24.444028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952941+0000","last_change":"2023-11-24T06:20:22.648994+0000","last_active":"2023-11-24T06:20:23.952941+0000","last_peered":"2023-11-24T06:20:23.952941+0000","last_clean":"2023-11-24T06:20:23.952941+0000","last_became_active":"2023-11-24T06:20:22.648752+0000","last_became_peered":"2023-11-24T06:20:22.648752+0000","last_unstale":"2023-11-24T06:20:23.952941+0000","last_undegraded":"2023-11-24T06:20:23.952941+0000","last_fullsized":"2023-11-24T06:20:23.952941+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:38.775210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952577+0000","last_change":"2023-11-24T06:20:22.650231+0000","last_active":"2023-11-24T06:20:23.952577+0000","last_peered":"2023-11-24T06:20:23.952577+0000","last_clean":"2023-11-24T06:20:23.952577+0000","last_became_active":"2023-11-24T06:20:22.649966+0000","last_became_peered":"2023-11-24T06:20:22.649966+0000","last_unstale":"2023-11-24T06:20:23.952577+0000","last_undegraded":"2023-11-24T06:20:23.952577+0000","last_fullsized":"2023-11-24T06:20:23.952577+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:56:33.904287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949212+0000","last_change":"2023-11-24T06:20:22.653260+0000","last_active":"2023-11-24T06:20:23.949212+0000","last_peered":"2023-11-24T06:20:23.949212+0000","last_clean":"2023-11-24T06:20:23.949212+0000","last_became_active":"2023-11-24T06:20:22.652992+0000","last_became_peered":"2023-11-24T06:20:22.652992+0000","last_unstale":"2023-11-24T06:20:23.949212+0000","last_undegraded":"2023-11-24T06:20:23.949212+0000","last_fullsized":"2023-11-24T06:20:23.949212+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:00:07.213948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952215+0000","last_change":"2023-11-24T06:20:22.653821+0000","last_active":"2023-11-24T06:20:23.952215+0000","last_peered":"2023-11-24T06:20:23.952215+0000","last_clean":"2023-11-24T06:20:23.952215+0000","last_became_active":"2023-11-24T06:20:22.651097+0000","last_became_peered":"2023-11-24T06:20:22.651097+0000","last_unstale":"2023-11-24T06:20:23.952215+0000","last_undegraded":"2023-11-24T06:20:23.952215+0000","last_fullsized":"2023-11-24T06:20:23.952215+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:38:42.017508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653307+0000","last_change":"2023-11-24T06:20:22.641192+0000","last_active":"2023-11-24T06:20:23.653307+0000","last_peered":"2023-11-24T06:20:23.653307+0000","last_clean":"2023-11-24T06:20:23.653307+0000","last_became_active":"2023-11-24T06:20:22.640908+0000","last_became_peered":"2023-11-24T06:20:22.640908+0000","last_unstale":"2023-11-24T06:20:23.653307+0000","last_undegraded":"2023-11-24T06:20:23.653307+0000","last_fullsized":"2023-11-24T06:20:23.653307+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:54.221321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949057+0000","last_change":"2023-11-24T06:20:22.651687+0000","last_active":"2023-11-24T06:20:23.949057+0000","last_peered":"2023-11-24T06:20:23.949057+0000","last_clean":"2023-11-24T06:20:23.949057+0000","last_became_active":"2023-11-24T06:20:22.651405+0000","last_became_peered":"2023-11-24T06:20:22.651405+0000","last_unstale":"2023-11-24T06:20:23.949057+0000","last_undegraded":"2023-11-24T06:20:23.949057+0000","last_fullsized":"2023-11-24T06:20:23.949057+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:53:36.733664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952024+0000","last_change":"2023-11-24T06:20:22.656048+0000","last_active":"2023-11-24T06:20:23.952024+0000","last_peered":"2023-11-24T06:20:23.952024+0000","last_clean":"2023-11-24T06:20:23.952024+0000","last_became_active":"2023-11-24T06:20:22.655680+0000","last_became_peered":"2023-11-24T06:20:22.655680+0000","last_unstale":"2023-11-24T06:20:23.952024+0000","last_undegraded":"2023-11-24T06:20:23.952024+0000","last_fullsized":"2023-11-24T06:20:23.952024+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:44.909329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952392+0000","last_change":"2023-11-24T06:20:22.648853+0000","last_active":"2023-11-24T06:20:23.952392+0000","last_peered":"2023-11-24T06:20:23.952392+0000","last_clean":"2023-11-24T06:20:23.952392+0000","last_became_active":"2023-11-24T06:20:22.648622+0000","last_became_peered":"2023-11-24T06:20:22.648622+0000","last_unstale":"2023-11-24T06:20:23.952392+0000","last_undegraded":"2023-11-24T06:20:23.952392+0000","last_fullsized":"2023-11-24T06:20:23.952392+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:38.775210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"52'4","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.664616+0000","last_change":"2023-11-24T06:20:21.635404+0000","last_active":"2023-11-24T06:20:22.664616+0000","last_peered":"2023-11-24T06:20:22.664616+0000","last_clean":"2023-11-24T06:20:22.664616+0000","last_became_active":"2023-11-24T06:20:21.635136+0000","last_became_peered":"2023-11-24T06:20:21.635136+0000","last_unstale":"2023-11-24T06:20:22.664616+0000","last_undegraded":"2023-11-24T06:20:22.664616+0000","last_fullsized":"2023-11-24T06:20:22.664616+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:06:48.708161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652209+0000","last_change":"2023-11-24T06:20:22.644736+0000","last_active":"2023-11-24T06:20:23.652209+0000","last_peered":"2023-11-24T06:20:23.652209+0000","last_clean":"2023-11-24T06:20:23.652209+0000","last_became_active":"2023-11-24T06:20:22.644303+0000","last_became_peered":"2023-11-24T06:20:22.644303+0000","last_unstale":"2023-11-24T06:20:23.652209+0000","last_undegraded":"2023-11-24T06:20:23.652209+0000","last_fullsized":"2023-11-24T06:20:23.652209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:02:32.900382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.661321+0000","last_change":"2023-11-24T06:20:21.634801+0000","last_active":"2023-11-24T06:20:22.661321+0000","last_peered":"2023-11-24T06:20:22.661321+0000","last_clean":"2023-11-24T06:20:22.661321+0000","last_became_active":"2023-11-24T06:20:21.634620+0000","last_became_peered":"2023-11-24T06:20:21.634620+0000","last_unstale":"2023-11-24T06:20:22.661321+0000","last_undegraded":"2023-11-24T06:20:22.661321+0000","last_fullsized":"2023-11-24T06:20:22.661321+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:09:35.477457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952397+0000","last_change":"2023-11-24T06:20:22.653000+0000","last_active":"2023-11-24T06:20:23.952397+0000","last_peered":"2023-11-24T06:20:23.952397+0000","last_clean":"2023-11-24T06:20:23.952397+0000","last_became_active":"2023-11-24T06:20:22.652709+0000","last_became_peered":"2023-11-24T06:20:22.652709+0000","last_unstale":"2023-11-24T06:20:23.952397+0000","last_undegraded":"2023-11-24T06:20:23.952397+0000","last_fullsized":"2023-11-24T06:20:23.952397+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:43:50.378588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.666296+0000","last_change":"2023-11-24T06:20:21.641256+0000","last_active":"2023-11-24T06:20:22.666296+0000","last_peered":"2023-11-24T06:20:22.666296+0000","last_clean":"2023-11-24T06:20:22.666296+0000","last_became_active":"2023-11-24T06:20:21.640930+0000","last_became_peered":"2023-11-24T06:20:21.640930+0000","last_unstale":"2023-11-24T06:20:22.666296+0000","last_undegraded":"2023-11-24T06:20:22.666296+0000","last_fullsized":"2023-11-24T06:20:22.666296+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952687+0000","last_change":"2023-11-24T06:20:22.651166+0000","last_active":"2023-11-24T06:20:23.952687+0000","last_peered":"2023-11-24T06:20:23.952687+0000","last_clean":"2023-11-24T06:20:23.952687+0000","last_became_active":"2023-11-24T06:20:22.650900+0000","last_became_peered":"2023-11-24T06:20:22.650900+0000","last_unstale":"2023-11-24T06:20:23.952687+0000","last_undegraded":"2023-11-24T06:20:23.952687+0000","last_fullsized":"2023-11-24T06:20:23.952687+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:36:36.186702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.660598+0000","last_change":"2023-11-24T06:20:21.634844+0000","last_active":"2023-11-24T06:20:22.660598+0000","last_peered":"2023-11-24T06:20:22.660598+0000","last_clean":"2023-11-24T06:20:22.660598+0000","last_became_active":"2023-11-24T06:20:21.634564+0000","last_became_peered":"2023-11-24T06:20:21.634564+0000","last_unstale":"2023-11-24T06:20:22.660598+0000","last_undegraded":"2023-11-24T06:20:22.660598+0000","last_fullsized":"2023-11-24T06:20:22.660598+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:23.554792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952666+0000","last_change":"2023-11-24T06:20:22.654286+0000","last_active":"2023-11-24T06:20:23.952666+0000","last_peered":"2023-11-24T06:20:23.952666+0000","last_clean":"2023-11-24T06:20:23.952666+0000","last_became_active":"2023-11-24T06:20:22.653575+0000","last_became_peered":"2023-11-24T06:20:22.653575+0000","last_unstale":"2023-11-24T06:20:23.952666+0000","last_undegraded":"2023-11-24T06:20:23.952666+0000","last_fullsized":"2023-11-24T06:20:23.952666+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:00.137286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663104+0000","last_change":"2023-11-24T06:20:21.629599+0000","last_active":"2023-11-24T06:20:22.663104+0000","last_peered":"2023-11-24T06:20:22.663104+0000","last_clean":"2023-11-24T06:20:22.663104+0000","last_became_active":"2023-11-24T06:20:21.629219+0000","last_became_peered":"2023-11-24T06:20:21.629219+0000","last_unstale":"2023-11-24T06:20:22.663104+0000","last_undegraded":"2023-11-24T06:20:22.663104+0000","last_fullsized":"2023-11-24T06:20:22.663104+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952356+0000","last_change":"2023-11-24T06:20:22.656353+0000","last_active":"2023-11-24T06:20:23.952356+0000","last_peered":"2023-11-24T06:20:23.952356+0000","last_clean":"2023-11-24T06:20:23.952356+0000","last_became_active":"2023-11-24T06:20:22.651935+0000","last_became_peered":"2023-11-24T06:20:22.651935+0000","last_unstale":"2023-11-24T06:20:23.952356+0000","last_undegraded":"2023-11-24T06:20:23.952356+0000","last_fullsized":"2023-11-24T06:20:23.952356+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:58:51.562896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663214+0000","last_change":"2023-11-24T06:20:21.635521+0000","last_active":"2023-11-24T06:20:22.663214+0000","last_peered":"2023-11-24T06:20:22.663214+0000","last_clean":"2023-11-24T06:20:22.663214+0000","last_became_active":"2023-11-24T06:20:21.635214+0000","last_became_peered":"2023-11-24T06:20:21.635214+0000","last_unstale":"2023-11-24T06:20:22.663214+0000","last_undegraded":"2023-11-24T06:20:22.663214+0000","last_fullsized":"2023-11-24T06:20:22.663214+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:24:57.040169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952805+0000","last_change":"2023-11-24T06:20:22.645140+0000","last_active":"2023-11-24T06:20:23.952805+0000","last_peered":"2023-11-24T06:20:23.952805+0000","last_clean":"2023-11-24T06:20:23.952805+0000","last_became_active":"2023-11-24T06:20:22.644851+0000","last_became_peered":"2023-11-24T06:20:22.644851+0000","last_unstale":"2023-11-24T06:20:23.952805+0000","last_undegraded":"2023-11-24T06:20:23.952805+0000","last_fullsized":"2023-11-24T06:20:23.952805+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:21.844277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.642268+0000","last_change":"2023-11-24T06:20:21.634192+0000","last_active":"2023-11-24T06:20:22.642268+0000","last_peered":"2023-11-24T06:20:22.642268+0000","last_clean":"2023-11-24T06:20:22.642268+0000","last_became_active":"2023-11-24T06:20:21.633818+0000","last_became_peered":"2023-11-24T06:20:21.633818+0000","last_unstale":"2023-11-24T06:20:22.642268+0000","last_undegraded":"2023-11-24T06:20:22.642268+0000","last_fullsized":"2023-11-24T06:20:22.642268+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952393+0000","last_change":"2023-11-24T06:20:22.651614+0000","last_active":"2023-11-24T06:20:23.952393+0000","last_peered":"2023-11-24T06:20:23.952393+0000","last_clean":"2023-11-24T06:20:23.952393+0000","last_became_active":"2023-11-24T06:20:22.650968+0000","last_became_peered":"2023-11-24T06:20:22.650968+0000","last_unstale":"2023-11-24T06:20:23.952393+0000","last_undegraded":"2023-11-24T06:20:23.952393+0000","last_fullsized":"2023-11-24T06:20:23.952393+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:45:55.663001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.633199+0000","last_change":"2023-11-24T06:20:21.637491+0000","last_active":"2023-11-24T06:20:22.633199+0000","last_peered":"2023-11-24T06:20:22.633199+0000","last_clean":"2023-11-24T06:20:22.633199+0000","last_became_active":"2023-11-24T06:20:21.637173+0000","last_became_peered":"2023-11-24T06:20:21.637173+0000","last_unstale":"2023-11-24T06:20:22.633199+0000","last_undegraded":"2023-11-24T06:20:22.633199+0000","last_fullsized":"2023-11-24T06:20:22.633199+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:23.435615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653310+0000","last_change":"2023-11-24T06:20:22.641823+0000","last_active":"2023-11-24T06:20:23.653310+0000","last_peered":"2023-11-24T06:20:23.653310+0000","last_clean":"2023-11-24T06:20:23.653310+0000","last_became_active":"2023-11-24T06:20:22.640647+0000","last_became_peered":"2023-11-24T06:20:22.640647+0000","last_unstale":"2023-11-24T06:20:23.653310+0000","last_undegraded":"2023-11-24T06:20:23.653310+0000","last_fullsized":"2023-11-24T06:20:23.653310+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:52.339933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.633682+0000","last_change":"2023-11-24T06:20:21.638447+0000","last_active":"2023-11-24T06:20:22.633682+0000","last_peered":"2023-11-24T06:20:22.633682+0000","last_clean":"2023-11-24T06:20:22.633682+0000","last_became_active":"2023-11-24T06:20:21.638158+0000","last_became_peered":"2023-11-24T06:20:21.638158+0000","last_unstale":"2023-11-24T06:20:22.633682+0000","last_undegraded":"2023-11-24T06:20:22.633682+0000","last_fullsized":"2023-11-24T06:20:22.633682+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.654106+0000","last_change":"2023-11-24T06:20:22.637249+0000","last_active":"2023-11-24T06:20:23.654106+0000","last_peered":"2023-11-24T06:20:23.654106+0000","last_clean":"2023-11-24T06:20:23.654106+0000","last_became_active":"2023-11-24T06:20:22.637004+0000","last_became_peered":"2023-11-24T06:20:22.637004+0000","last_unstale":"2023-11-24T06:20:23.654106+0000","last_undegraded":"2023-11-24T06:20:23.654106+0000","last_fullsized":"2023-11-24T06:20:23.654106+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:21:52.740150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.642128+0000","last_change":"2023-11-24T06:20:21.631051+0000","last_active":"2023-11-24T06:20:22.642128+0000","last_peered":"2023-11-24T06:20:22.642128+0000","last_clean":"2023-11-24T06:20:22.642128+0000","last_became_active":"2023-11-24T06:20:21.630773+0000","last_became_peered":"2023-11-24T06:20:21.630773+0000","last_unstale":"2023-11-24T06:20:22.642128+0000","last_undegraded":"2023-11-24T06:20:22.642128+0000","last_fullsized":"2023-11-24T06:20:22.642128+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:40.450161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652085+0000","last_change":"2023-11-24T06:20:22.644895+0000","last_active":"2023-11-24T06:20:23.652085+0000","last_peered":"2023-11-24T06:20:23.652085+0000","last_clean":"2023-11-24T06:20:23.652085+0000","last_became_active":"2023-11-24T06:20:22.644442+0000","last_became_peered":"2023-11-24T06:20:22.644442+0000","last_unstale":"2023-11-24T06:20:23.652085+0000","last_undegraded":"2023-11-24T06:20:23.652085+0000","last_fullsized":"2023-11-24T06:20:23.652085+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:24:58.044711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"52'3","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.662485+0000","last_change":"2023-11-24T06:20:21.637050+0000","last_active":"2023-11-24T06:20:22.662485+0000","last_peered":"2023-11-24T06:20:22.662485+0000","last_clean":"2023-11-24T06:20:22.662485+0000","last_became_active":"2023-11-24T06:20:21.636769+0000","last_became_peered":"2023-11-24T06:20:21.636769+0000","last_unstale":"2023-11-24T06:20:22.662485+0000","last_undegraded":"2023-11-24T06:20:22.662485+0000","last_fullsized":"2023-11-24T06:20:22.662485+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:37.770668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653499+0000","last_change":"2023-11-24T06:20:22.644841+0000","last_active":"2023-11-24T06:20:23.653499+0000","last_peered":"2023-11-24T06:20:23.653499+0000","last_clean":"2023-11-24T06:20:23.653499+0000","last_became_active":"2023-11-24T06:20:22.644425+0000","last_became_peered":"2023-11-24T06:20:22.644425+0000","last_unstale":"2023-11-24T06:20:23.653499+0000","last_undegraded":"2023-11-24T06:20:23.653499+0000","last_fullsized":"2023-11-24T06:20:23.653499+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:48:46.545922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"52'6","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.666376+0000","last_change":"2023-11-24T06:20:21.638561+0000","last_active":"2023-11-24T06:20:22.666376+0000","last_peered":"2023-11-24T06:20:22.666376+0000","last_clean":"2023-11-24T06:20:22.666376+0000","last_became_active":"2023-11-24T06:20:21.638252+0000","last_became_peered":"2023-11-24T06:20:21.638252+0000","last_unstale":"2023-11-24T06:20:22.666376+0000","last_undegraded":"2023-11-24T06:20:22.666376+0000","last_fullsized":"2023-11-24T06:20:22.666376+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:54:05.115719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952940+0000","last_change":"2023-11-24T06:20:22.651874+0000","last_active":"2023-11-24T06:20:23.952940+0000","last_peered":"2023-11-24T06:20:23.952940+0000","last_clean":"2023-11-24T06:20:23.952940+0000","last_became_active":"2023-11-24T06:20:22.651478+0000","last_became_peered":"2023-11-24T06:20:22.651478+0000","last_unstale":"2023-11-24T06:20:23.952940+0000","last_undegraded":"2023-11-24T06:20:23.952940+0000","last_fullsized":"2023-11-24T06:20:23.952940+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:45.138650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663836+0000","last_change":"2023-11-24T06:20:21.633330+0000","last_active":"2023-11-24T06:20:22.663836+0000","last_peered":"2023-11-24T06:20:22.663836+0000","last_clean":"2023-11-24T06:20:22.663836+0000","last_became_active":"2023-11-24T06:20:21.633057+0000","last_became_peered":"2023-11-24T06:20:21.633057+0000","last_unstale":"2023-11-24T06:20:22.663836+0000","last_undegraded":"2023-11-24T06:20:22.663836+0000","last_fullsized":"2023-11-24T06:20:22.663836+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:27:19.915683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653602+0000","last_change":"2023-11-24T06:20:22.647188+0000","last_active":"2023-11-24T06:20:23.653602+0000","last_peered":"2023-11-24T06:20:23.653602+0000","last_clean":"2023-11-24T06:20:23.653602+0000","last_became_active":"2023-11-24T06:20:22.646928+0000","last_became_peered":"2023-11-24T06:20:22.646928+0000","last_unstale":"2023-11-24T06:20:23.653602+0000","last_undegraded":"2023-11-24T06:20:23.653602+0000","last_fullsized":"2023-11-24T06:20:23.653602+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:51:46.249755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.639163+0000","last_change":"2023-11-24T06:20:21.632526+0000","last_active":"2023-11-24T06:20:22.639163+0000","last_peered":"2023-11-24T06:20:22.639163+0000","last_clean":"2023-11-24T06:20:22.639163+0000","last_became_active":"2023-11-24T06:20:21.632218+0000","last_became_peered":"2023-11-24T06:20:21.632218+0000","last_unstale":"2023-11-24T06:20:22.639163+0000","last_undegraded":"2023-11-24T06:20:22.639163+0000","last_fullsized":"2023-11-24T06:20:22.639163+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:02:30.419332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652788+0000","last_change":"2023-11-24T06:20:22.638687+0000","last_active":"2023-11-24T06:20:23.652788+0000","last_peered":"2023-11-24T06:20:23.652788+0000","last_clean":"2023-11-24T06:20:23.652788+0000","last_became_active":"2023-11-24T06:20:22.638428+0000","last_became_peered":"2023-11-24T06:20:22.638428+0000","last_unstale":"2023-11-24T06:20:23.652788+0000","last_undegraded":"2023-11-24T06:20:23.652788+0000","last_fullsized":"2023-11-24T06:20:23.652788+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:35:46.501400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.642406+0000","last_change":"2023-11-24T06:20:21.635534+0000","last_active":"2023-11-24T06:20:22.642406+0000","last_peered":"2023-11-24T06:20:22.642406+0000","last_clean":"2023-11-24T06:20:22.642406+0000","last_became_active":"2023-11-24T06:20:21.635099+0000","last_became_peered":"2023-11-24T06:20:21.635099+0000","last_unstale":"2023-11-24T06:20:22.642406+0000","last_undegraded":"2023-11-24T06:20:22.642406+0000","last_fullsized":"2023-11-24T06:20:22.642406+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:26.240404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653408+0000","last_change":"2023-11-24T06:20:22.655319+0000","last_active":"2023-11-24T06:20:23.653408+0000","last_peered":"2023-11-24T06:20:23.653408+0000","last_clean":"2023-11-24T06:20:23.653408+0000","last_became_active":"2023-11-24T06:20:22.654992+0000","last_became_peered":"2023-11-24T06:20:22.654992+0000","last_unstale":"2023-11-24T06:20:23.653408+0000","last_undegraded":"2023-11-24T06:20:23.653408+0000","last_fullsized":"2023-11-24T06:20:23.653408+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:09:36.482000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"52'8","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.664518+0000","last_change":"2023-11-24T06:20:21.634951+0000","last_active":"2023-11-24T06:20:22.664518+0000","last_peered":"2023-11-24T06:20:22.664518+0000","last_clean":"2023-11-24T06:20:22.664518+0000","last_became_active":"2023-11-24T06:20:21.634653+0000","last_became_peered":"2023-11-24T06:20:21.634653+0000","last_unstale":"2023-11-24T06:20:22.664518+0000","last_undegraded":"2023-11-24T06:20:22.664518+0000","last_fullsized":"2023-11-24T06:20:22.664518+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:34.255885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.654141+0000","last_change":"2023-11-24T06:20:22.641352+0000","last_active":"2023-11-24T06:20:23.654141+0000","last_peered":"2023-11-24T06:20:23.654141+0000","last_clean":"2023-11-24T06:20:23.654141+0000","last_became_active":"2023-11-24T06:20:22.641069+0000","last_became_peered":"2023-11-24T06:20:22.641069+0000","last_unstale":"2023-11-24T06:20:23.654141+0000","last_undegraded":"2023-11-24T06:20:23.654141+0000","last_fullsized":"2023-11-24T06:20:23.654141+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:17:17.159648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.661270+0000","last_change":"2023-11-24T06:20:21.633653+0000","last_active":"2023-11-24T06:20:22.661270+0000","last_peered":"2023-11-24T06:20:22.661270+0000","last_clean":"2023-11-24T06:20:22.661270+0000","last_became_active":"2023-11-24T06:20:21.633333+0000","last_became_peered":"2023-11-24T06:20:21.633333+0000","last_unstale":"2023-11-24T06:20:22.661270+0000","last_undegraded":"2023-11-24T06:20:22.661270+0000","last_fullsized":"2023-11-24T06:20:22.661270+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953077+0000","last_change":"2023-11-24T06:20:22.651717+0000","last_active":"2023-11-24T06:20:23.953077+0000","last_peered":"2023-11-24T06:20:23.953077+0000","last_clean":"2023-11-24T06:20:23.953077+0000","last_became_active":"2023-11-24T06:20:22.651384+0000","last_became_peered":"2023-11-24T06:20:22.651384+0000","last_unstale":"2023-11-24T06:20:23.953077+0000","last_undegraded":"2023-11-24T06:20:23.953077+0000","last_fullsized":"2023-11-24T06:20:23.953077+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:43:50.378588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.664283+0000","last_change":"2023-11-24T06:20:21.638281+0000","last_active":"2023-11-24T06:20:22.664283+0000","last_peered":"2023-11-24T06:20:22.664283+0000","last_clean":"2023-11-24T06:20:22.664283+0000","last_became_active":"2023-11-24T06:20:21.638014+0000","last_became_peered":"2023-11-24T06:20:21.638014+0000","last_unstale":"2023-11-24T06:20:22.664283+0000","last_undegraded":"2023-11-24T06:20:22.664283+0000","last_fullsized":"2023-11-24T06:20:22.664283+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:34.255885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952990+0000","last_change":"2023-11-24T06:20:22.648184+0000","last_active":"2023-11-24T06:20:23.952990+0000","last_peered":"2023-11-24T06:20:23.952990+0000","last_clean":"2023-11-24T06:20:23.952990+0000","last_became_active":"2023-11-24T06:20:22.647911+0000","last_became_peered":"2023-11-24T06:20:22.647911+0000","last_unstale":"2023-11-24T06:20:23.952990+0000","last_undegraded":"2023-11-24T06:20:23.952990+0000","last_fullsized":"2023-11-24T06:20:23.952990+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:08.604050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663097+0000","last_change":"2023-11-24T06:20:21.634745+0000","last_active":"2023-11-24T06:20:22.663097+0000","last_peered":"2023-11-24T06:20:22.663097+0000","last_clean":"2023-11-24T06:20:22.663097+0000","last_became_active":"2023-11-24T06:20:21.634453+0000","last_became_peered":"2023-11-24T06:20:21.634453+0000","last_unstale":"2023-11-24T06:20:22.663097+0000","last_undegraded":"2023-11-24T06:20:22.663097+0000","last_fullsized":"2023-11-24T06:20:22.663097+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:34.255885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653519+0000","last_change":"2023-11-24T06:20:22.638288+0000","last_active":"2023-11-24T06:20:23.653519+0000","last_peered":"2023-11-24T06:20:23.653519+0000","last_clean":"2023-11-24T06:20:23.653519+0000","last_became_active":"2023-11-24T06:20:22.638019+0000","last_became_peered":"2023-11-24T06:20:22.638019+0000","last_unstale":"2023-11-24T06:20:23.653519+0000","last_undegraded":"2023-11-24T06:20:23.653519+0000","last_fullsized":"2023-11-24T06:20:23.653519+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:55.130880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.640208+0000","last_change":"2023-11-24T06:20:21.640109+0000","last_active":"2023-11-24T06:20:22.640208+0000","last_peered":"2023-11-24T06:20:22.640208+0000","last_clean":"2023-11-24T06:20:22.640208+0000","last_became_active":"2023-11-24T06:20:21.639777+0000","last_became_peered":"2023-11-24T06:20:21.639777+0000","last_unstale":"2023-11-24T06:20:22.640208+0000","last_undegraded":"2023-11-24T06:20:22.640208+0000","last_fullsized":"2023-11-24T06:20:22.640208+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:30.394419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953139+0000","last_change":"2023-11-24T06:20:22.651732+0000","last_active":"2023-11-24T06:20:23.953139+0000","last_peered":"2023-11-24T06:20:23.953139+0000","last_clean":"2023-11-24T06:20:23.953139+0000","last_became_active":"2023-11-24T06:20:22.651227+0000","last_became_peered":"2023-11-24T06:20:22.651227+0000","last_unstale":"2023-11-24T06:20:23.953139+0000","last_undegraded":"2023-11-24T06:20:23.953139+0000","last_fullsized":"2023-11-24T06:20:23.953139+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:43:02.349069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663397+0000","last_change":"2023-11-24T06:20:21.631857+0000","last_active":"2023-11-24T06:20:22.663397+0000","last_peered":"2023-11-24T06:20:22.663397+0000","last_clean":"2023-11-24T06:20:22.663397+0000","last_became_active":"2023-11-24T06:20:21.631396+0000","last_became_peered":"2023-11-24T06:20:21.631396+0000","last_unstale":"2023-11-24T06:20:22.663397+0000","last_undegraded":"2023-11-24T06:20:22.663397+0000","last_fullsized":"2023-11-24T06:20:22.663397+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:12:22.460606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952386+0000","last_change":"2023-11-24T06:20:22.648369+0000","last_active":"2023-11-24T06:20:23.952386+0000","last_peered":"2023-11-24T06:20:23.952386+0000","last_clean":"2023-11-24T06:20:23.952386+0000","last_became_active":"2023-11-24T06:20:22.647963+0000","last_became_peered":"2023-11-24T06:20:22.647963+0000","last_unstale":"2023-11-24T06:20:23.952386+0000","last_undegraded":"2023-11-24T06:20:23.952386+0000","last_fullsized":"2023-11-24T06:20:23.952386+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:30.081700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.634136+0000","last_change":"2023-11-24T06:20:21.633654+0000","last_active":"2023-11-24T06:20:22.634136+0000","last_peered":"2023-11-24T06:20:22.634136+0000","last_clean":"2023-11-24T06:20:22.634136+0000","last_became_active":"2023-11-24T06:20:21.633387+0000","last_became_peered":"2023-11-24T06:20:21.633387+0000","last_unstale":"2023-11-24T06:20:22.634136+0000","last_undegraded":"2023-11-24T06:20:22.634136+0000","last_fullsized":"2023-11-24T06:20:22.634136+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:24.432980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952097+0000","last_change":"2023-11-24T06:20:22.646935+0000","last_active":"2023-11-24T06:20:23.952097+0000","last_peered":"2023-11-24T06:20:23.952097+0000","last_clean":"2023-11-24T06:20:23.952097+0000","last_became_active":"2023-11-24T06:20:22.646663+0000","last_became_peered":"2023-11-24T06:20:22.646663+0000","last_unstale":"2023-11-24T06:20:23.952097+0000","last_undegraded":"2023-11-24T06:20:23.952097+0000","last_fullsized":"2023-11-24T06:20:23.952097+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:27.244946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.641390+0000","last_change":"2023-11-24T06:20:21.639872+0000","last_active":"2023-11-24T06:20:22.641390+0000","last_peered":"2023-11-24T06:20:22.641390+0000","last_clean":"2023-11-24T06:20:22.641390+0000","last_became_active":"2023-11-24T06:20:21.639600+0000","last_became_peered":"2023-11-24T06:20:21.639600+0000","last_unstale":"2023-11-24T06:20:22.641390+0000","last_undegraded":"2023-11-24T06:20:22.641390+0000","last_fullsized":"2023-11-24T06:20:22.641390+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:13:34.095847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953092+0000","last_change":"2023-11-24T06:20:22.652562+0000","last_active":"2023-11-24T06:20:23.953092+0000","last_peered":"2023-11-24T06:20:23.953092+0000","last_clean":"2023-11-24T06:20:23.953092+0000","last_became_active":"2023-11-24T06:20:22.651592+0000","last_became_peered":"2023-11-24T06:20:22.651592+0000","last_unstale":"2023-11-24T06:20:23.953092+0000","last_undegraded":"2023-11-24T06:20:23.953092+0000","last_fullsized":"2023-11-24T06:20:23.953092+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:36:36.186702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.662133+0000","last_change":"2023-11-24T06:20:21.638221+0000","last_active":"2023-11-24T06:20:22.662133+0000","last_peered":"2023-11-24T06:20:22.662133+0000","last_clean":"2023-11-24T06:20:22.662133+0000","last_became_active":"2023-11-24T06:20:21.637800+0000","last_became_peered":"2023-11-24T06:20:21.637800+0000","last_unstale":"2023-11-24T06:20:22.662133+0000","last_undegraded":"2023-11-24T06:20:22.662133+0000","last_fullsized":"2023-11-24T06:20:22.662133+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952951+0000","last_change":"2023-11-24T06:20:22.648346+0000","last_active":"2023-11-24T06:20:23.952951+0000","last_peered":"2023-11-24T06:20:23.952951+0000","last_clean":"2023-11-24T06:20:23.952951+0000","last_became_active":"2023-11-24T06:20:22.646016+0000","last_became_peered":"2023-11-24T06:20:22.646016+0000","last_unstale":"2023-11-24T06:20:23.952951+0000","last_undegraded":"2023-11-24T06:20:23.952951+0000","last_fullsized":"2023-11-24T06:20:23.952951+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:30.081700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.641445+0000","last_change":"2023-11-24T06:20:21.635897+0000","last_active":"2023-11-24T06:20:22.641445+0000","last_peered":"2023-11-24T06:20:22.641445+0000","last_clean":"2023-11-24T06:20:22.641445+0000","last_became_active":"2023-11-24T06:20:21.635167+0000","last_became_peered":"2023-11-24T06:20:21.635167+0000","last_unstale":"2023-11-24T06:20:22.641445+0000","last_undegraded":"2023-11-24T06:20:22.641445+0000","last_fullsized":"2023-11-24T06:20:22.641445+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:13:34.095847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652227+0000","last_change":"2023-11-24T06:20:22.644709+0000","last_active":"2023-11-24T06:20:23.652227+0000","last_peered":"2023-11-24T06:20:23.652227+0000","last_clean":"2023-11-24T06:20:23.652227+0000","last_became_active":"2023-11-24T06:20:22.644112+0000","last_became_peered":"2023-11-24T06:20:22.644112+0000","last_unstale":"2023-11-24T06:20:23.652227+0000","last_undegraded":"2023-11-24T06:20:23.652227+0000","last_fullsized":"2023-11-24T06:20:23.652227+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:31.398961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.641110+0000","last_change":"2023-11-24T06:20:21.633952+0000","last_active":"2023-11-24T06:20:22.641110+0000","last_peered":"2023-11-24T06:20:22.641110+0000","last_clean":"2023-11-24T06:20:22.641110+0000","last_became_active":"2023-11-24T06:20:21.633629+0000","last_became_peered":"2023-11-24T06:20:21.633629+0000","last_unstale":"2023-11-24T06:20:22.641110+0000","last_undegraded":"2023-11-24T06:20:22.641110+0000","last_fullsized":"2023-11-24T06:20:22.641110+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:17:23.296026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952770+0000","last_change":"2023-11-24T06:20:22.653035+0000","last_active":"2023-11-24T06:20:23.952770+0000","last_peered":"2023-11-24T06:20:23.952770+0000","last_clean":"2023-11-24T06:20:23.952770+0000","last_became_active":"2023-11-24T06:20:22.652698+0000","last_became_peered":"2023-11-24T06:20:22.652698+0000","last_unstale":"2023-11-24T06:20:23.952770+0000","last_undegraded":"2023-11-24T06:20:23.952770+0000","last_fullsized":"2023-11-24T06:20:23.952770+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:43:02.349069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.642278+0000","last_change":"2023-11-24T06:20:21.634883+0000","last_active":"2023-11-24T06:20:22.642278+0000","last_peered":"2023-11-24T06:20:22.642278+0000","last_clean":"2023-11-24T06:20:22.642278+0000","last_became_active":"2023-11-24T06:20:21.634624+0000","last_became_peered":"2023-11-24T06:20:21.634624+0000","last_unstale":"2023-11-24T06:20:22.642278+0000","last_undegraded":"2023-11-24T06:20:22.642278+0000","last_fullsized":"2023-11-24T06:20:22.642278+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:43.904787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952660+0000","last_change":"2023-11-24T06:20:22.656329+0000","last_active":"2023-11-24T06:20:23.952660+0000","last_peered":"2023-11-24T06:20:23.952660+0000","last_clean":"2023-11-24T06:20:23.952660+0000","last_became_active":"2023-11-24T06:20:22.652087+0000","last_became_peered":"2023-11-24T06:20:22.652087+0000","last_unstale":"2023-11-24T06:20:23.952660+0000","last_undegraded":"2023-11-24T06:20:23.952660+0000","last_fullsized":"2023-11-24T06:20:23.952660+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:38:40.237128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.635011+0000","last_change":"2023-11-24T06:20:21.635390+0000","last_active":"2023-11-24T06:20:22.635011+0000","last_peered":"2023-11-24T06:20:22.635011+0000","last_clean":"2023-11-24T06:20:22.635011+0000","last_became_active":"2023-11-24T06:20:21.635114+0000","last_became_peered":"2023-11-24T06:20:21.635114+0000","last_unstale":"2023-11-24T06:20:22.635011+0000","last_undegraded":"2023-11-24T06:20:22.635011+0000","last_fullsized":"2023-11-24T06:20:22.635011+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:18:34.137206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653684+0000","last_change":"2023-11-24T06:20:22.646535+0000","last_active":"2023-11-24T06:20:23.653684+0000","last_peered":"2023-11-24T06:20:23.653684+0000","last_clean":"2023-11-24T06:20:23.653684+0000","last_became_active":"2023-11-24T06:20:22.646260+0000","last_became_peered":"2023-11-24T06:20:22.646260+0000","last_unstale":"2023-11-24T06:20:23.653684+0000","last_undegraded":"2023-11-24T06:20:23.653684+0000","last_fullsized":"2023-11-24T06:20:23.653684+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:41.454704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.633444+0000","last_change":"2023-11-24T06:20:21.636357+0000","last_active":"2023-11-24T06:20:22.633444+0000","last_peered":"2023-11-24T06:20:22.633444+0000","last_clean":"2023-11-24T06:20:22.633444+0000","last_became_active":"2023-11-24T06:20:21.636068+0000","last_became_peered":"2023-11-24T06:20:21.636068+0000","last_unstale":"2023-11-24T06:20:22.633444+0000","last_undegraded":"2023-11-24T06:20:22.633444+0000","last_fullsized":"2023-11-24T06:20:22.633444+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:38.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.651753+0000","last_change":"2023-11-24T06:20:22.646637+0000","last_active":"2023-11-24T06:20:23.651753+0000","last_peered":"2023-11-24T06:20:23.651753+0000","last_clean":"2023-11-24T06:20:23.651753+0000","last_became_active":"2023-11-24T06:20:22.646370+0000","last_became_peered":"2023-11-24T06:20:22.646370+0000","last_unstale":"2023-11-24T06:20:23.651753+0000","last_undegraded":"2023-11-24T06:20:23.651753+0000","last_fullsized":"2023-11-24T06:20:23.651753+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:16.523717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.634140+0000","last_change":"2023-11-24T06:20:21.641031+0000","last_active":"2023-11-24T06:20:22.634140+0000","last_peered":"2023-11-24T06:20:22.634140+0000","last_clean":"2023-11-24T06:20:22.634140+0000","last_became_active":"2023-11-24T06:20:21.640826+0000","last_became_peered":"2023-11-24T06:20:21.640826+0000","last_unstale":"2023-11-24T06:20:22.634140+0000","last_undegraded":"2023-11-24T06:20:22.634140+0000","last_fullsized":"2023-11-24T06:20:22.634140+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:25.850351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952522+0000","last_change":"2023-11-24T06:20:22.651781+0000","last_active":"2023-11-24T06:20:23.952522+0000","last_peered":"2023-11-24T06:20:23.952522+0000","last_clean":"2023-11-24T06:20:23.952522+0000","last_became_active":"2023-11-24T06:20:22.651361+0000","last_became_peered":"2023-11-24T06:20:22.651361+0000","last_unstale":"2023-11-24T06:20:23.952522+0000","last_undegraded":"2023-11-24T06:20:23.952522+0000","last_fullsized":"2023-11-24T06:20:23.952522+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:15:09.954871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.640147+0000","last_change":"2023-11-24T06:20:21.636646+0000","last_active":"2023-11-24T06:20:22.640147+0000","last_peered":"2023-11-24T06:20:22.640147+0000","last_clean":"2023-11-24T06:20:22.640147+0000","last_became_active":"2023-11-24T06:20:21.636335+0000","last_became_peered":"2023-11-24T06:20:21.636335+0000","last_unstale":"2023-11-24T06:20:22.640147+0000","last_undegraded":"2023-11-24T06:20:22.640147+0000","last_fullsized":"2023-11-24T06:20:22.640147+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:53.216779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952434+0000","last_change":"2023-11-24T06:20:22.653597+0000","last_active":"2023-11-24T06:20:23.952434+0000","last_peered":"2023-11-24T06:20:23.952434+0000","last_clean":"2023-11-24T06:20:23.952434+0000","last_became_active":"2023-11-24T06:20:22.651007+0000","last_became_peered":"2023-11-24T06:20:22.651007+0000","last_unstale":"2023-11-24T06:20:23.952434+0000","last_undegraded":"2023-11-24T06:20:23.952434+0000","last_fullsized":"2023-11-24T06:20:23.952434+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:03:16.942755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.638921+0000","last_change":"2023-11-24T06:20:21.635744+0000","last_active":"2023-11-24T06:20:22.638921+0000","last_peered":"2023-11-24T06:20:22.638921+0000","last_clean":"2023-11-24T06:20:22.638921+0000","last_became_active":"2023-11-24T06:20:21.635447+0000","last_became_peered":"2023-11-24T06:20:21.635447+0000","last_unstale":"2023-11-24T06:20:22.638921+0000","last_undegraded":"2023-11-24T06:20:22.638921+0000","last_fullsized":"2023-11-24T06:20:22.638921+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:21.256791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.633273+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:22.633273+0000","last_peered":"2023-11-24T06:20:22.633273+0000","last_clean":"2023-11-24T06:20:22.633273+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:22.633273+0000","last_undegraded":"2023-11-24T06:20:22.633273+0000","last_fullsized":"2023-11-24T06:20:22.633273+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28228,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343612,"statfs":{"total":96636764160,"available":96607858688,"internally_reserved":0,"allocated":1236992,"data_stored":860946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60599999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.65300000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.71199999999999997}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.504},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.623},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67100000000000004}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.755},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.63800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":270418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":6.423,"5min":6.423,"15min":6.423},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.505}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.46899999999999997}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":12.554,"5min":12.554,"15min":12.554},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.625},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.65500000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":270508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.311}]},{"osd":3,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.29199999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.62},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":862980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.441}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.42399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.77200000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.72099999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.88900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":270474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.308},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.374}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.753}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.71399999999999997}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.82599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":270474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.27000000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.40000000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.58799999999999997}]},{"osd":5,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.77000000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.621}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":272030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.44500000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.751}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":271070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.04,"5min":3.04,"15min":3.04},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.621},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.501}]},{"osd":3,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":10.198,"5min":10.198,"15min":10.198},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":10.215999999999999,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.66400000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.70099999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.64800000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:20:28.611 INFO:tasks.ceph.mds.c.smithi089.stderr:2023-11-24T06:20:28.608+0000 7f681759b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:20:31.320 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:20:31.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.452+0000 7f69214fd700 1 Processor -- start 2023-11-24T06:20:31.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.452+0000 7f69214fd700 1 -- start start 2023-11-24T06:20:31.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.452+0000 7f69214fd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f691c15e480 0x7f691c0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.452+0000 7f69214fd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c0d3610 0x7f691c0b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.452+0000 7f69214fd700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0b62e0 con 0x7f691c0b5e90 2023-11-24T06:20:31.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.452+0000 7f69214fd700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c0d2a70 con 0x7f691c15e480 2023-11-24T06:20:31.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.452+0000 7f69214fd700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0d2d50 con 0x7f691c0d3610 2023-11-24T06:20:31.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691affd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f691c15e480 0x7f691c0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691affd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f691c15e480 0x7f691c0d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38048/0 (socket says 172.21.15.89:38048) 2023-11-24T06:20:31.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691affd700 1 -- 172.21.15.89:0/3603338221 learned_addr learned my addr 172.21.15.89:0/3603338221 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691a7fc700 1 --2- 172.21.15.89:0/3603338221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c0d3610 0x7f691c0b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691affd700 1 -- 172.21.15.89:0/3603338221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c0d3610 msgr2=0x7f691c0b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691affd700 1 --2- 172.21.15.89:0/3603338221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c0d3610 0x7f691c0b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691affd700 1 -- 172.21.15.89:0/3603338221 >> v1:172.21.15.123:6789/0 conn(0x7f691c0b5e90 legacy=0x7f691c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691affd700 1 -- 172.21.15.89:0/3603338221 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0d3030 con 0x7f691c15e480 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f691affd700 1 --2- 172.21.15.89:0/3603338221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c15e480 0x7f691c0d2100 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f69100098f0 tx=0x7f6910009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1744e5ab7b717f98 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f6919ffb700 1 -- 172.21.15.89:0/3603338221 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910011830 con 0x7f691c15e480 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f6919ffb700 1 -- 172.21.15.89:0/3603338221 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6910011990 con 0x7f691c15e480 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.453+0000 7f6919ffb700 1 -- 172.21.15.89:0/3603338221 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910011d20 con 0x7f691c15e480 2023-11-24T06:20:31.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.454+0000 7f69214fd700 1 -- 172.21.15.89:0/3603338221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c15e480 msgr2=0x7f691c0d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.454+0000 7f69214fd700 1 --2- 172.21.15.89:0/3603338221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c15e480 0x7f691c0d2100 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f69100098f0 tx=0x7f6910009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:31.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.454+0000 7f69214fd700 1 -- 172.21.15.89:0/3603338221 shutdown_connections 2023-11-24T06:20:31.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.454+0000 7f69214fd700 1 --2- 172.21.15.89:0/3603338221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c15e480 0x7f691c0d2100 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.454+0000 7f69214fd700 1 --2- 172.21.15.89:0/3603338221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c0d3610 0x7f691c0b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.454+0000 7f69214fd700 1 -- 172.21.15.89:0/3603338221 >> 172.21.15.89:0/3603338221 conn(0x7f691c0d0950 msgr2=0x7f691c0d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:31.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.454+0000 7f69214fd700 1 -- 172.21.15.89:0/3603338221 shutdown_connections 2023-11-24T06:20:31.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.454+0000 7f69214fd700 1 -- 172.21.15.89:0/3603338221 wait complete. 2023-11-24T06:20:31.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.455+0000 7f69214fd700 1 Processor -- start 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.455+0000 7f69214fd700 1 -- start start 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f69214fd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c0b5e90 0x7f691c0c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f69214fd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f691c0d3610 0x7f691c151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f69214fd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c15e480 0x7f691c06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f69214fd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0c9c40 con 0x7f691c0b5e90 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f69214fd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0c9680 con 0x7f691c0d3610 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f69214fd700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0c9960 con 0x7f691c15e480 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c0b5e90 0x7f691c0c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c0b5e90 0x7f691c0c88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38052/0 (socket says 172.21.15.89:38052) 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 -- 172.21.15.89:0/2458276729 learned_addr learned my addr 172.21.15.89:0/2458276729 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 -- 172.21.15.89:0/2458276729 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c15e480 msgr2=0x7f691c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c15e480 0x7f691c06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 -- 172.21.15.89:0/2458276729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f691c0d3610 msgr2=0x7f691c151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691a7fc700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f691c0d3610 0x7f691c151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f691c0d3610 0x7f691c151420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 -- 172.21.15.89:0/2458276729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6910009550 con 0x7f691c0b5e90 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.456+0000 7f691affd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c0b5e90 0x7f691c0c88f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6910009c40 tx=0x7f6910004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.457+0000 7f69037fe700 1 -- 172.21.15.89:0/2458276729 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69100036a0 con 0x7f691c0b5e90 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.457+0000 7f69037fe700 1 -- 172.21.15.89:0/2458276729 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691001e070 con 0x7f691c0b5e90 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.457+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c15fa50 con 0x7f691c0b5e90 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.457+0000 7f69037fe700 1 -- 172.21.15.89:0/2458276729 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910003a40 con 0x7f691c0b5e90 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.457+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c0c8e30 con 0x7f691c0b5e90 2023-11-24T06:20:31.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.458+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c15f8b0 con 0x7f691c0b5e90 2023-11-24T06:20:31.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.458+0000 7f69037fe700 1 -- 172.21.15.89:0/2458276729 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6910003ba0 con 0x7f691c0b5e90 2023-11-24T06:20:31.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.458+0000 7f69037fe700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f690806aed0 0x7f690806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.458+0000 7f69037fe700 1 -- 172.21.15.89:0/2458276729 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f691008a510 con 0x7f691c0b5e90 2023-11-24T06:20:31.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.458+0000 7f691a7fc700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f690806aed0 0x7f690806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.459+0000 7f691a7fc700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f690806aed0 0x7f690806d390 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f691c0b3890 tx=0x7f690c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:31.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.463+0000 7f69037fe700 1 -- 172.21.15.89:0/2458276729 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f691c15f8b0 con 0x7f691c0b5e90 2023-11-24T06:20:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.649+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f691c0d2640 con 0x7f690806aed0 2023-11-24T06:20:31.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.657+0000 7f69037fe700 1 -- 172.21.15.89:0/2458276729 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289502 (secure 0 0 0) 0x7f690c0106b0 con 0x7f690806aed0 2023-11-24T06:20:31.663 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:31.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f690806aed0 msgr2=0x7f690806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f690806aed0 0x7f690806d390 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f691c0b3890 tx=0x7f690c008040 comp rx=0 tx=0).stop 2023-11-24T06:20:31.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c0b5e90 msgr2=0x7f691c0c88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c0b5e90 0x7f691c0c88f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6910009c40 tx=0x7f6910004750 comp rx=0 tx=0).stop 2023-11-24T06:20:31.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 shutdown_connections 2023-11-24T06:20:31.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f691c15e480 0x7f691c06aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f691c0b5e90 0x7f691c0c88f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f690806aed0 0x7f690806d390 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 --2- 172.21.15.89:0/2458276729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f691c0d3610 0x7f691c151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 >> 172.21.15.89:0/2458276729 conn(0x7f691c0d0950 msgr2=0x7f691c0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:31.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 shutdown_connections 2023-11-24T06:20:31.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.660+0000 7f69214fd700 1 -- 172.21.15.89:0/2458276729 wait complete. 2023-11-24T06:20:31.666 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:20:31.675 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2023-11-24T06:20:29.739863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222368,"kb_used_data":6184,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752352,"statfs":{"total":773094113280,"available":772866408448,"internally_reserved":0,"allocated":6332416,"data_stored":3348900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001856"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949353+0000","last_change":"2023-11-24T06:20:22.653131+0000","last_active":"2023-11-24T06:20:23.949353+0000","last_peered":"2023-11-24T06:20:23.949353+0000","last_clean":"2023-11-24T06:20:23.949353+0000","last_became_active":"2023-11-24T06:20:22.652875+0000","last_became_peered":"2023-11-24T06:20:22.652875+0000","last_unstale":"2023-11-24T06:20:23.949353+0000","last_undegraded":"2023-11-24T06:20:23.949353+0000","last_fullsized":"2023-11-24T06:20:23.949353+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:28:41.054467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953004+0000","last_change":"2023-11-24T06:20:22.656133+0000","last_active":"2023-11-24T06:20:23.953004+0000","last_peered":"2023-11-24T06:20:23.953004+0000","last_clean":"2023-11-24T06:20:23.953004+0000","last_became_active":"2023-11-24T06:20:22.655721+0000","last_became_peered":"2023-11-24T06:20:22.655721+0000","last_unstale":"2023-11-24T06:20:23.953004+0000","last_undegraded":"2023-11-24T06:20:23.953004+0000","last_fullsized":"2023-11-24T06:20:23.953004+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:03.760844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653343+0000","last_change":"2023-11-24T06:20:22.642395+0000","last_active":"2023-11-24T06:20:23.653343+0000","last_peered":"2023-11-24T06:20:23.653343+0000","last_clean":"2023-11-24T06:20:23.653343+0000","last_became_active":"2023-11-24T06:20:22.642101+0000","last_became_peered":"2023-11-24T06:20:22.642101+0000","last_unstale":"2023-11-24T06:20:23.653343+0000","last_undegraded":"2023-11-24T06:20:23.653343+0000","last_fullsized":"2023-11-24T06:20:23.653343+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:05.174311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953108+0000","last_change":"2023-11-24T06:20:22.652050+0000","last_active":"2023-11-24T06:20:23.953108+0000","last_peered":"2023-11-24T06:20:23.953108+0000","last_clean":"2023-11-24T06:20:23.953108+0000","last_became_active":"2023-11-24T06:20:22.651804+0000","last_became_peered":"2023-11-24T06:20:22.651804+0000","last_unstale":"2023-11-24T06:20:23.953108+0000","last_undegraded":"2023-11-24T06:20:23.953108+0000","last_fullsized":"2023-11-24T06:20:23.953108+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:56:33.904287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949240+0000","last_change":"2023-11-24T06:20:22.651948+0000","last_active":"2023-11-24T06:20:23.949240+0000","last_peered":"2023-11-24T06:20:23.949240+0000","last_clean":"2023-11-24T06:20:23.949240+0000","last_became_active":"2023-11-24T06:20:22.651653+0000","last_became_peered":"2023-11-24T06:20:22.651653+0000","last_unstale":"2023-11-24T06:20:23.949240+0000","last_undegraded":"2023-11-24T06:20:23.949240+0000","last_fullsized":"2023-11-24T06:20:23.949240+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:06.512497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952159+0000","last_change":"2023-11-24T06:20:22.653183+0000","last_active":"2023-11-24T06:20:23.952159+0000","last_peered":"2023-11-24T06:20:23.952159+0000","last_clean":"2023-11-24T06:20:23.952159+0000","last_became_active":"2023-11-24T06:20:22.652787+0000","last_became_peered":"2023-11-24T06:20:22.652787+0000","last_unstale":"2023-11-24T06:20:23.952159+0000","last_undegraded":"2023-11-24T06:20:23.952159+0000","last_fullsized":"2023-11-24T06:20:23.952159+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:45.138650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652864+0000","last_change":"2023-11-24T06:20:22.643286+0000","last_active":"2023-11-24T06:20:23.652864+0000","last_peered":"2023-11-24T06:20:23.652864+0000","last_clean":"2023-11-24T06:20:23.652864+0000","last_became_active":"2023-11-24T06:20:22.642994+0000","last_became_peered":"2023-11-24T06:20:22.642994+0000","last_unstale":"2023-11-24T06:20:23.652864+0000","last_undegraded":"2023-11-24T06:20:23.652864+0000","last_fullsized":"2023-11-24T06:20:23.652864+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:02:32.900382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952340+0000","last_change":"2023-11-24T06:20:22.648090+0000","last_active":"2023-11-24T06:20:23.952340+0000","last_peered":"2023-11-24T06:20:23.952340+0000","last_clean":"2023-11-24T06:20:23.952340+0000","last_became_active":"2023-11-24T06:20:22.647796+0000","last_became_peered":"2023-11-24T06:20:22.647796+0000","last_unstale":"2023-11-24T06:20:23.952340+0000","last_undegraded":"2023-11-24T06:20:23.952340+0000","last_fullsized":"2023-11-24T06:20:23.952340+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:03.457321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653032+0000","last_change":"2023-11-24T06:20:22.651119+0000","last_active":"2023-11-24T06:20:23.653032+0000","last_peered":"2023-11-24T06:20:23.653032+0000","last_clean":"2023-11-24T06:20:23.653032+0000","last_became_active":"2023-11-24T06:20:22.650793+0000","last_became_peered":"2023-11-24T06:20:22.650793+0000","last_unstale":"2023-11-24T06:20:23.653032+0000","last_undegraded":"2023-11-24T06:20:23.653032+0000","last_fullsized":"2023-11-24T06:20:23.653032+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:31.398961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652093+0000","last_change":"2023-11-24T06:20:22.645989+0000","last_active":"2023-11-24T06:20:23.652093+0000","last_peered":"2023-11-24T06:20:23.652093+0000","last_clean":"2023-11-24T06:20:23.652093+0000","last_became_active":"2023-11-24T06:20:22.645725+0000","last_became_peered":"2023-11-24T06:20:22.645725+0000","last_unstale":"2023-11-24T06:20:23.652093+0000","last_undegraded":"2023-11-24T06:20:23.652093+0000","last_fullsized":"2023-11-24T06:20:23.652093+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:35:46.501400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652229+0000","last_change":"2023-11-24T06:20:22.654068+0000","last_active":"2023-11-24T06:20:23.652229+0000","last_peered":"2023-11-24T06:20:23.652229+0000","last_clean":"2023-11-24T06:20:23.652229+0000","last_became_active":"2023-11-24T06:20:22.653765+0000","last_became_peered":"2023-11-24T06:20:22.653765+0000","last_unstale":"2023-11-24T06:20:23.652229+0000","last_undegraded":"2023-11-24T06:20:23.652229+0000","last_fullsized":"2023-11-24T06:20:23.652229+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:04:43.643723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953347+0000","last_change":"2023-11-24T06:20:22.648555+0000","last_active":"2023-11-24T06:20:23.953347+0000","last_peered":"2023-11-24T06:20:23.953347+0000","last_clean":"2023-11-24T06:20:23.953347+0000","last_became_active":"2023-11-24T06:20:22.646184+0000","last_became_peered":"2023-11-24T06:20:22.646184+0000","last_unstale":"2023-11-24T06:20:23.953347+0000","last_undegraded":"2023-11-24T06:20:23.953347+0000","last_fullsized":"2023-11-24T06:20:23.953347+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:18.409593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953412+0000","last_change":"2023-11-24T06:20:22.649705+0000","last_active":"2023-11-24T06:20:23.953412+0000","last_peered":"2023-11-24T06:20:23.953412+0000","last_clean":"2023-11-24T06:20:23.953412+0000","last_became_active":"2023-11-24T06:20:22.649413+0000","last_became_peered":"2023-11-24T06:20:22.649413+0000","last_unstale":"2023-11-24T06:20:23.953412+0000","last_undegraded":"2023-11-24T06:20:23.953412+0000","last_fullsized":"2023-11-24T06:20:23.953412+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:43:24.451689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653311+0000","last_change":"2023-11-24T06:20:22.650989+0000","last_active":"2023-11-24T06:20:23.653311+0000","last_peered":"2023-11-24T06:20:23.653311+0000","last_clean":"2023-11-24T06:20:23.653311+0000","last_became_active":"2023-11-24T06:20:22.650747+0000","last_became_peered":"2023-11-24T06:20:22.650747+0000","last_unstale":"2023-11-24T06:20:23.653311+0000","last_undegraded":"2023-11-24T06:20:23.653311+0000","last_fullsized":"2023-11-24T06:20:23.653311+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:41:46.472796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652932+0000","last_change":"2023-11-24T06:20:22.641140+0000","last_active":"2023-11-24T06:20:23.652932+0000","last_peered":"2023-11-24T06:20:23.652932+0000","last_clean":"2023-11-24T06:20:23.652932+0000","last_became_active":"2023-11-24T06:20:22.640830+0000","last_became_peered":"2023-11-24T06:20:22.640830+0000","last_unstale":"2023-11-24T06:20:23.652932+0000","last_undegraded":"2023-11-24T06:20:23.652932+0000","last_fullsized":"2023-11-24T06:20:23.652932+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:04:43.643723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952163+0000","last_change":"2023-11-24T06:20:22.646028+0000","last_active":"2023-11-24T06:20:23.952163+0000","last_peered":"2023-11-24T06:20:23.952163+0000","last_clean":"2023-11-24T06:20:23.952163+0000","last_became_active":"2023-11-24T06:20:22.645701+0000","last_became_peered":"2023-11-24T06:20:22.645701+0000","last_unstale":"2023-11-24T06:20:23.952163+0000","last_undegraded":"2023-11-24T06:20:23.952163+0000","last_fullsized":"2023-11-24T06:20:23.952163+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:20.360413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953148+0000","last_change":"2023-11-24T06:20:22.651149+0000","last_active":"2023-11-24T06:20:23.953148+0000","last_peered":"2023-11-24T06:20:23.953148+0000","last_clean":"2023-11-24T06:20:23.953148+0000","last_became_active":"2023-11-24T06:20:22.650825+0000","last_became_peered":"2023-11-24T06:20:22.650825+0000","last_unstale":"2023-11-24T06:20:23.953148+0000","last_undegraded":"2023-11-24T06:20:23.953148+0000","last_fullsized":"2023-11-24T06:20:23.953148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:32:25.860603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.654044+0000","last_change":"2023-11-24T06:20:22.650034+0000","last_active":"2023-11-24T06:20:23.654044+0000","last_peered":"2023-11-24T06:20:23.654044+0000","last_clean":"2023-11-24T06:20:23.654044+0000","last_became_active":"2023-11-24T06:20:22.649540+0000","last_became_peered":"2023-11-24T06:20:22.649540+0000","last_unstale":"2023-11-24T06:20:23.654044+0000","last_undegraded":"2023-11-24T06:20:23.654044+0000","last_fullsized":"2023-11-24T06:20:23.654044+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:11.136219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949211+0000","last_change":"2023-11-24T06:20:22.647946+0000","last_active":"2023-11-24T06:20:23.949211+0000","last_peered":"2023-11-24T06:20:23.949211+0000","last_clean":"2023-11-24T06:20:23.949211+0000","last_became_active":"2023-11-24T06:20:22.647680+0000","last_became_peered":"2023-11-24T06:20:22.647680+0000","last_unstale":"2023-11-24T06:20:23.949211+0000","last_undegraded":"2023-11-24T06:20:23.949211+0000","last_fullsized":"2023-11-24T06:20:23.949211+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:11:37.991620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952928+0000","last_change":"2023-11-24T06:20:22.648917+0000","last_active":"2023-11-24T06:20:23.952928+0000","last_peered":"2023-11-24T06:20:23.952928+0000","last_clean":"2023-11-24T06:20:23.952928+0000","last_became_active":"2023-11-24T06:20:22.648656+0000","last_became_peered":"2023-11-24T06:20:22.648656+0000","last_unstale":"2023-11-24T06:20:23.952928+0000","last_undegraded":"2023-11-24T06:20:23.952928+0000","last_fullsized":"2023-11-24T06:20:23.952928+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:56:08.700328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653517+0000","last_change":"2023-11-24T06:20:22.645991+0000","last_active":"2023-11-24T06:20:23.653517+0000","last_peered":"2023-11-24T06:20:23.653517+0000","last_clean":"2023-11-24T06:20:23.653517+0000","last_became_active":"2023-11-24T06:20:22.645726+0000","last_became_peered":"2023-11-24T06:20:22.645726+0000","last_unstale":"2023-11-24T06:20:23.653517+0000","last_undegraded":"2023-11-24T06:20:23.653517+0000","last_fullsized":"2023-11-24T06:20:23.653517+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:28:18.701104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952528+0000","last_change":"2023-11-24T06:20:22.654435+0000","last_active":"2023-11-24T06:20:23.952528+0000","last_peered":"2023-11-24T06:20:23.952528+0000","last_clean":"2023-11-24T06:20:23.952528+0000","last_became_active":"2023-11-24T06:20:22.653999+0000","last_became_peered":"2023-11-24T06:20:22.653999+0000","last_unstale":"2023-11-24T06:20:23.952528+0000","last_undegraded":"2023-11-24T06:20:23.952528+0000","last_fullsized":"2023-11-24T06:20:23.952528+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:54.507295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653240+0000","last_change":"2023-11-24T06:20:22.652948+0000","last_active":"2023-11-24T06:20:23.653240+0000","last_peered":"2023-11-24T06:20:23.653240+0000","last_clean":"2023-11-24T06:20:23.653240+0000","last_became_active":"2023-11-24T06:20:22.652706+0000","last_became_peered":"2023-11-24T06:20:22.652706+0000","last_unstale":"2023-11-24T06:20:23.653240+0000","last_undegraded":"2023-11-24T06:20:23.653240+0000","last_fullsized":"2023-11-24T06:20:23.653240+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:18.409593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952649+0000","last_change":"2023-11-24T06:20:22.654236+0000","last_active":"2023-11-24T06:20:23.952649+0000","last_peered":"2023-11-24T06:20:23.952649+0000","last_clean":"2023-11-24T06:20:23.952649+0000","last_became_active":"2023-11-24T06:20:22.654019+0000","last_became_peered":"2023-11-24T06:20:22.654019+0000","last_unstale":"2023-11-24T06:20:23.952649+0000","last_undegraded":"2023-11-24T06:20:23.952649+0000","last_fullsized":"2023-11-24T06:20:23.952649+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:06.068523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949341+0000","last_change":"2023-11-24T06:20:22.651422+0000","last_active":"2023-11-24T06:20:23.949341+0000","last_peered":"2023-11-24T06:20:23.949341+0000","last_clean":"2023-11-24T06:20:23.949341+0000","last_became_active":"2023-11-24T06:20:22.651140+0000","last_became_peered":"2023-11-24T06:20:22.651140+0000","last_unstale":"2023-11-24T06:20:23.949341+0000","last_undegraded":"2023-11-24T06:20:23.949341+0000","last_fullsized":"2023-11-24T06:20:23.949341+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:57:24.444028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952941+0000","last_change":"2023-11-24T06:20:22.648994+0000","last_active":"2023-11-24T06:20:23.952941+0000","last_peered":"2023-11-24T06:20:23.952941+0000","last_clean":"2023-11-24T06:20:23.952941+0000","last_became_active":"2023-11-24T06:20:22.648752+0000","last_became_peered":"2023-11-24T06:20:22.648752+0000","last_unstale":"2023-11-24T06:20:23.952941+0000","last_undegraded":"2023-11-24T06:20:23.952941+0000","last_fullsized":"2023-11-24T06:20:23.952941+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:38.775210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952577+0000","last_change":"2023-11-24T06:20:22.650231+0000","last_active":"2023-11-24T06:20:23.952577+0000","last_peered":"2023-11-24T06:20:23.952577+0000","last_clean":"2023-11-24T06:20:23.952577+0000","last_became_active":"2023-11-24T06:20:22.649966+0000","last_became_peered":"2023-11-24T06:20:22.649966+0000","last_unstale":"2023-11-24T06:20:23.952577+0000","last_undegraded":"2023-11-24T06:20:23.952577+0000","last_fullsized":"2023-11-24T06:20:23.952577+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:56:33.904287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949212+0000","last_change":"2023-11-24T06:20:22.653260+0000","last_active":"2023-11-24T06:20:23.949212+0000","last_peered":"2023-11-24T06:20:23.949212+0000","last_clean":"2023-11-24T06:20:23.949212+0000","last_became_active":"2023-11-24T06:20:22.652992+0000","last_became_peered":"2023-11-24T06:20:22.652992+0000","last_unstale":"2023-11-24T06:20:23.949212+0000","last_undegraded":"2023-11-24T06:20:23.949212+0000","last_fullsized":"2023-11-24T06:20:23.949212+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:00:07.213948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952215+0000","last_change":"2023-11-24T06:20:22.653821+0000","last_active":"2023-11-24T06:20:23.952215+0000","last_peered":"2023-11-24T06:20:23.952215+0000","last_clean":"2023-11-24T06:20:23.952215+0000","last_became_active":"2023-11-24T06:20:22.651097+0000","last_became_peered":"2023-11-24T06:20:22.651097+0000","last_unstale":"2023-11-24T06:20:23.952215+0000","last_undegraded":"2023-11-24T06:20:23.952215+0000","last_fullsized":"2023-11-24T06:20:23.952215+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:38:42.017508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653307+0000","last_change":"2023-11-24T06:20:22.641192+0000","last_active":"2023-11-24T06:20:23.653307+0000","last_peered":"2023-11-24T06:20:23.653307+0000","last_clean":"2023-11-24T06:20:23.653307+0000","last_became_active":"2023-11-24T06:20:22.640908+0000","last_became_peered":"2023-11-24T06:20:22.640908+0000","last_unstale":"2023-11-24T06:20:23.653307+0000","last_undegraded":"2023-11-24T06:20:23.653307+0000","last_fullsized":"2023-11-24T06:20:23.653307+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:54.221321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.949057+0000","last_change":"2023-11-24T06:20:22.651687+0000","last_active":"2023-11-24T06:20:23.949057+0000","last_peered":"2023-11-24T06:20:23.949057+0000","last_clean":"2023-11-24T06:20:23.949057+0000","last_became_active":"2023-11-24T06:20:22.651405+0000","last_became_peered":"2023-11-24T06:20:22.651405+0000","last_unstale":"2023-11-24T06:20:23.949057+0000","last_undegraded":"2023-11-24T06:20:23.949057+0000","last_fullsized":"2023-11-24T06:20:23.949057+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:53:36.733664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952024+0000","last_change":"2023-11-24T06:20:22.656048+0000","last_active":"2023-11-24T06:20:23.952024+0000","last_peered":"2023-11-24T06:20:23.952024+0000","last_clean":"2023-11-24T06:20:23.952024+0000","last_became_active":"2023-11-24T06:20:22.655680+0000","last_became_peered":"2023-11-24T06:20:22.655680+0000","last_unstale":"2023-11-24T06:20:23.952024+0000","last_undegraded":"2023-11-24T06:20:23.952024+0000","last_fullsized":"2023-11-24T06:20:23.952024+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:44.909329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952392+0000","last_change":"2023-11-24T06:20:22.648853+0000","last_active":"2023-11-24T06:20:23.952392+0000","last_peered":"2023-11-24T06:20:23.952392+0000","last_clean":"2023-11-24T06:20:23.952392+0000","last_became_active":"2023-11-24T06:20:22.648622+0000","last_became_peered":"2023-11-24T06:20:22.648622+0000","last_unstale":"2023-11-24T06:20:23.952392+0000","last_undegraded":"2023-11-24T06:20:23.952392+0000","last_fullsized":"2023-11-24T06:20:23.952392+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:38.775210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"52'4","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.664616+0000","last_change":"2023-11-24T06:20:21.635404+0000","last_active":"2023-11-24T06:20:22.664616+0000","last_peered":"2023-11-24T06:20:22.664616+0000","last_clean":"2023-11-24T06:20:22.664616+0000","last_became_active":"2023-11-24T06:20:21.635136+0000","last_became_peered":"2023-11-24T06:20:21.635136+0000","last_unstale":"2023-11-24T06:20:22.664616+0000","last_undegraded":"2023-11-24T06:20:22.664616+0000","last_fullsized":"2023-11-24T06:20:22.664616+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:06:48.708161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652209+0000","last_change":"2023-11-24T06:20:22.644736+0000","last_active":"2023-11-24T06:20:23.652209+0000","last_peered":"2023-11-24T06:20:23.652209+0000","last_clean":"2023-11-24T06:20:23.652209+0000","last_became_active":"2023-11-24T06:20:22.644303+0000","last_became_peered":"2023-11-24T06:20:22.644303+0000","last_unstale":"2023-11-24T06:20:23.652209+0000","last_undegraded":"2023-11-24T06:20:23.652209+0000","last_fullsized":"2023-11-24T06:20:23.652209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:02:32.900382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.661321+0000","last_change":"2023-11-24T06:20:21.634801+0000","last_active":"2023-11-24T06:20:22.661321+0000","last_peered":"2023-11-24T06:20:22.661321+0000","last_clean":"2023-11-24T06:20:22.661321+0000","last_became_active":"2023-11-24T06:20:21.634620+0000","last_became_peered":"2023-11-24T06:20:21.634620+0000","last_unstale":"2023-11-24T06:20:22.661321+0000","last_undegraded":"2023-11-24T06:20:22.661321+0000","last_fullsized":"2023-11-24T06:20:22.661321+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:09:35.477457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952397+0000","last_change":"2023-11-24T06:20:22.653000+0000","last_active":"2023-11-24T06:20:23.952397+0000","last_peered":"2023-11-24T06:20:23.952397+0000","last_clean":"2023-11-24T06:20:23.952397+0000","last_became_active":"2023-11-24T06:20:22.652709+0000","last_became_peered":"2023-11-24T06:20:22.652709+0000","last_unstale":"2023-11-24T06:20:23.952397+0000","last_undegraded":"2023-11-24T06:20:23.952397+0000","last_fullsized":"2023-11-24T06:20:23.952397+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:43:50.378588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.666296+0000","last_change":"2023-11-24T06:20:21.641256+0000","last_active":"2023-11-24T06:20:22.666296+0000","last_peered":"2023-11-24T06:20:22.666296+0000","last_clean":"2023-11-24T06:20:22.666296+0000","last_became_active":"2023-11-24T06:20:21.640930+0000","last_became_peered":"2023-11-24T06:20:21.640930+0000","last_unstale":"2023-11-24T06:20:22.666296+0000","last_undegraded":"2023-11-24T06:20:22.666296+0000","last_fullsized":"2023-11-24T06:20:22.666296+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952687+0000","last_change":"2023-11-24T06:20:22.651166+0000","last_active":"2023-11-24T06:20:23.952687+0000","last_peered":"2023-11-24T06:20:23.952687+0000","last_clean":"2023-11-24T06:20:23.952687+0000","last_became_active":"2023-11-24T06:20:22.650900+0000","last_became_peered":"2023-11-24T06:20:22.650900+0000","last_unstale":"2023-11-24T06:20:23.952687+0000","last_undegraded":"2023-11-24T06:20:23.952687+0000","last_fullsized":"2023-11-24T06:20:23.952687+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:36:36.186702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.660598+0000","last_change":"2023-11-24T06:20:21.634844+0000","last_active":"2023-11-24T06:20:22.660598+0000","last_peered":"2023-11-24T06:20:22.660598+0000","last_clean":"2023-11-24T06:20:22.660598+0000","last_became_active":"2023-11-24T06:20:21.634564+0000","last_became_peered":"2023-11-24T06:20:21.634564+0000","last_unstale":"2023-11-24T06:20:22.660598+0000","last_undegraded":"2023-11-24T06:20:22.660598+0000","last_fullsized":"2023-11-24T06:20:22.660598+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:23.554792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952666+0000","last_change":"2023-11-24T06:20:22.654286+0000","last_active":"2023-11-24T06:20:23.952666+0000","last_peered":"2023-11-24T06:20:23.952666+0000","last_clean":"2023-11-24T06:20:23.952666+0000","last_became_active":"2023-11-24T06:20:22.653575+0000","last_became_peered":"2023-11-24T06:20:22.653575+0000","last_unstale":"2023-11-24T06:20:23.952666+0000","last_undegraded":"2023-11-24T06:20:23.952666+0000","last_fullsized":"2023-11-24T06:20:23.952666+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:00.137286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663104+0000","last_change":"2023-11-24T06:20:21.629599+0000","last_active":"2023-11-24T06:20:22.663104+0000","last_peered":"2023-11-24T06:20:22.663104+0000","last_clean":"2023-11-24T06:20:22.663104+0000","last_became_active":"2023-11-24T06:20:21.629219+0000","last_became_peered":"2023-11-24T06:20:21.629219+0000","last_unstale":"2023-11-24T06:20:22.663104+0000","last_undegraded":"2023-11-24T06:20:22.663104+0000","last_fullsized":"2023-11-24T06:20:22.663104+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952356+0000","last_change":"2023-11-24T06:20:22.656353+0000","last_active":"2023-11-24T06:20:23.952356+0000","last_peered":"2023-11-24T06:20:23.952356+0000","last_clean":"2023-11-24T06:20:23.952356+0000","last_became_active":"2023-11-24T06:20:22.651935+0000","last_became_peered":"2023-11-24T06:20:22.651935+0000","last_unstale":"2023-11-24T06:20:23.952356+0000","last_undegraded":"2023-11-24T06:20:23.952356+0000","last_fullsized":"2023-11-24T06:20:23.952356+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:58:51.562896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663214+0000","last_change":"2023-11-24T06:20:21.635521+0000","last_active":"2023-11-24T06:20:22.663214+0000","last_peered":"2023-11-24T06:20:22.663214+0000","last_clean":"2023-11-24T06:20:22.663214+0000","last_became_active":"2023-11-24T06:20:21.635214+0000","last_became_peered":"2023-11-24T06:20:21.635214+0000","last_unstale":"2023-11-24T06:20:22.663214+0000","last_undegraded":"2023-11-24T06:20:22.663214+0000","last_fullsized":"2023-11-24T06:20:22.663214+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:24:57.040169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952805+0000","last_change":"2023-11-24T06:20:22.645140+0000","last_active":"2023-11-24T06:20:23.952805+0000","last_peered":"2023-11-24T06:20:23.952805+0000","last_clean":"2023-11-24T06:20:23.952805+0000","last_became_active":"2023-11-24T06:20:22.644851+0000","last_became_peered":"2023-11-24T06:20:22.644851+0000","last_unstale":"2023-11-24T06:20:23.952805+0000","last_undegraded":"2023-11-24T06:20:23.952805+0000","last_fullsized":"2023-11-24T06:20:23.952805+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:21.844277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.642268+0000","last_change":"2023-11-24T06:20:21.634192+0000","last_active":"2023-11-24T06:20:22.642268+0000","last_peered":"2023-11-24T06:20:22.642268+0000","last_clean":"2023-11-24T06:20:22.642268+0000","last_became_active":"2023-11-24T06:20:21.633818+0000","last_became_peered":"2023-11-24T06:20:21.633818+0000","last_unstale":"2023-11-24T06:20:22.642268+0000","last_undegraded":"2023-11-24T06:20:22.642268+0000","last_fullsized":"2023-11-24T06:20:22.642268+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952393+0000","last_change":"2023-11-24T06:20:22.651614+0000","last_active":"2023-11-24T06:20:23.952393+0000","last_peered":"2023-11-24T06:20:23.952393+0000","last_clean":"2023-11-24T06:20:23.952393+0000","last_became_active":"2023-11-24T06:20:22.650968+0000","last_became_peered":"2023-11-24T06:20:22.650968+0000","last_unstale":"2023-11-24T06:20:23.952393+0000","last_undegraded":"2023-11-24T06:20:23.952393+0000","last_fullsized":"2023-11-24T06:20:23.952393+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:45:55.663001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.633199+0000","last_change":"2023-11-24T06:20:21.637491+0000","last_active":"2023-11-24T06:20:22.633199+0000","last_peered":"2023-11-24T06:20:22.633199+0000","last_clean":"2023-11-24T06:20:22.633199+0000","last_became_active":"2023-11-24T06:20:21.637173+0000","last_became_peered":"2023-11-24T06:20:21.637173+0000","last_unstale":"2023-11-24T06:20:22.633199+0000","last_undegraded":"2023-11-24T06:20:22.633199+0000","last_fullsized":"2023-11-24T06:20:22.633199+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:07:23.435615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653310+0000","last_change":"2023-11-24T06:20:22.641823+0000","last_active":"2023-11-24T06:20:23.653310+0000","last_peered":"2023-11-24T06:20:23.653310+0000","last_clean":"2023-11-24T06:20:23.653310+0000","last_became_active":"2023-11-24T06:20:22.640647+0000","last_became_peered":"2023-11-24T06:20:22.640647+0000","last_unstale":"2023-11-24T06:20:23.653310+0000","last_undegraded":"2023-11-24T06:20:23.653310+0000","last_fullsized":"2023-11-24T06:20:23.653310+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:52.339933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.633682+0000","last_change":"2023-11-24T06:20:21.638447+0000","last_active":"2023-11-24T06:20:22.633682+0000","last_peered":"2023-11-24T06:20:22.633682+0000","last_clean":"2023-11-24T06:20:22.633682+0000","last_became_active":"2023-11-24T06:20:21.638158+0000","last_became_peered":"2023-11-24T06:20:21.638158+0000","last_unstale":"2023-11-24T06:20:22.633682+0000","last_undegraded":"2023-11-24T06:20:22.633682+0000","last_fullsized":"2023-11-24T06:20:22.633682+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:49:34.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.654106+0000","last_change":"2023-11-24T06:20:22.637249+0000","last_active":"2023-11-24T06:20:23.654106+0000","last_peered":"2023-11-24T06:20:23.654106+0000","last_clean":"2023-11-24T06:20:23.654106+0000","last_became_active":"2023-11-24T06:20:22.637004+0000","last_became_peered":"2023-11-24T06:20:22.637004+0000","last_unstale":"2023-11-24T06:20:23.654106+0000","last_undegraded":"2023-11-24T06:20:23.654106+0000","last_fullsized":"2023-11-24T06:20:23.654106+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:21:52.740150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.642128+0000","last_change":"2023-11-24T06:20:21.631051+0000","last_active":"2023-11-24T06:20:22.642128+0000","last_peered":"2023-11-24T06:20:22.642128+0000","last_clean":"2023-11-24T06:20:22.642128+0000","last_became_active":"2023-11-24T06:20:21.630773+0000","last_became_peered":"2023-11-24T06:20:21.630773+0000","last_unstale":"2023-11-24T06:20:22.642128+0000","last_undegraded":"2023-11-24T06:20:22.642128+0000","last_fullsized":"2023-11-24T06:20:22.642128+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:40.450161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652085+0000","last_change":"2023-11-24T06:20:22.644895+0000","last_active":"2023-11-24T06:20:23.652085+0000","last_peered":"2023-11-24T06:20:23.652085+0000","last_clean":"2023-11-24T06:20:23.652085+0000","last_became_active":"2023-11-24T06:20:22.644442+0000","last_became_peered":"2023-11-24T06:20:22.644442+0000","last_unstale":"2023-11-24T06:20:23.652085+0000","last_undegraded":"2023-11-24T06:20:23.652085+0000","last_fullsized":"2023-11-24T06:20:23.652085+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:24:58.044711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"52'3","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.662485+0000","last_change":"2023-11-24T06:20:21.637050+0000","last_active":"2023-11-24T06:20:22.662485+0000","last_peered":"2023-11-24T06:20:22.662485+0000","last_clean":"2023-11-24T06:20:22.662485+0000","last_became_active":"2023-11-24T06:20:21.636769+0000","last_became_peered":"2023-11-24T06:20:21.636769+0000","last_unstale":"2023-11-24T06:20:22.662485+0000","last_undegraded":"2023-11-24T06:20:22.662485+0000","last_fullsized":"2023-11-24T06:20:22.662485+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:56:37.770668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653499+0000","last_change":"2023-11-24T06:20:22.644841+0000","last_active":"2023-11-24T06:20:23.653499+0000","last_peered":"2023-11-24T06:20:23.653499+0000","last_clean":"2023-11-24T06:20:23.653499+0000","last_became_active":"2023-11-24T06:20:22.644425+0000","last_became_peered":"2023-11-24T06:20:22.644425+0000","last_unstale":"2023-11-24T06:20:23.653499+0000","last_undegraded":"2023-11-24T06:20:23.653499+0000","last_fullsized":"2023-11-24T06:20:23.653499+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:48:46.545922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"52'6","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.666376+0000","last_change":"2023-11-24T06:20:21.638561+0000","last_active":"2023-11-24T06:20:22.666376+0000","last_peered":"2023-11-24T06:20:22.666376+0000","last_clean":"2023-11-24T06:20:22.666376+0000","last_became_active":"2023-11-24T06:20:21.638252+0000","last_became_peered":"2023-11-24T06:20:21.638252+0000","last_unstale":"2023-11-24T06:20:22.666376+0000","last_undegraded":"2023-11-24T06:20:22.666376+0000","last_fullsized":"2023-11-24T06:20:22.666376+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:54:05.115719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952940+0000","last_change":"2023-11-24T06:20:22.651874+0000","last_active":"2023-11-24T06:20:23.952940+0000","last_peered":"2023-11-24T06:20:23.952940+0000","last_clean":"2023-11-24T06:20:23.952940+0000","last_became_active":"2023-11-24T06:20:22.651478+0000","last_became_peered":"2023-11-24T06:20:22.651478+0000","last_unstale":"2023-11-24T06:20:23.952940+0000","last_undegraded":"2023-11-24T06:20:23.952940+0000","last_fullsized":"2023-11-24T06:20:23.952940+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:23:45.138650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663836+0000","last_change":"2023-11-24T06:20:21.633330+0000","last_active":"2023-11-24T06:20:22.663836+0000","last_peered":"2023-11-24T06:20:22.663836+0000","last_clean":"2023-11-24T06:20:22.663836+0000","last_became_active":"2023-11-24T06:20:21.633057+0000","last_became_peered":"2023-11-24T06:20:21.633057+0000","last_unstale":"2023-11-24T06:20:22.663836+0000","last_undegraded":"2023-11-24T06:20:22.663836+0000","last_fullsized":"2023-11-24T06:20:22.663836+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:27:19.915683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653602+0000","last_change":"2023-11-24T06:20:22.647188+0000","last_active":"2023-11-24T06:20:23.653602+0000","last_peered":"2023-11-24T06:20:23.653602+0000","last_clean":"2023-11-24T06:20:23.653602+0000","last_became_active":"2023-11-24T06:20:22.646928+0000","last_became_peered":"2023-11-24T06:20:22.646928+0000","last_unstale":"2023-11-24T06:20:23.653602+0000","last_undegraded":"2023-11-24T06:20:23.653602+0000","last_fullsized":"2023-11-24T06:20:23.653602+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:51:46.249755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.639163+0000","last_change":"2023-11-24T06:20:21.632526+0000","last_active":"2023-11-24T06:20:22.639163+0000","last_peered":"2023-11-24T06:20:22.639163+0000","last_clean":"2023-11-24T06:20:22.639163+0000","last_became_active":"2023-11-24T06:20:21.632218+0000","last_became_peered":"2023-11-24T06:20:21.632218+0000","last_unstale":"2023-11-24T06:20:22.639163+0000","last_undegraded":"2023-11-24T06:20:22.639163+0000","last_fullsized":"2023-11-24T06:20:22.639163+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:02:30.419332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652788+0000","last_change":"2023-11-24T06:20:22.638687+0000","last_active":"2023-11-24T06:20:23.652788+0000","last_peered":"2023-11-24T06:20:23.652788+0000","last_clean":"2023-11-24T06:20:23.652788+0000","last_became_active":"2023-11-24T06:20:22.638428+0000","last_became_peered":"2023-11-24T06:20:22.638428+0000","last_unstale":"2023-11-24T06:20:23.652788+0000","last_undegraded":"2023-11-24T06:20:23.652788+0000","last_fullsized":"2023-11-24T06:20:23.652788+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:35:46.501400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.642406+0000","last_change":"2023-11-24T06:20:21.635534+0000","last_active":"2023-11-24T06:20:22.642406+0000","last_peered":"2023-11-24T06:20:22.642406+0000","last_clean":"2023-11-24T06:20:22.642406+0000","last_became_active":"2023-11-24T06:20:21.635099+0000","last_became_peered":"2023-11-24T06:20:21.635099+0000","last_unstale":"2023-11-24T06:20:22.642406+0000","last_undegraded":"2023-11-24T06:20:22.642406+0000","last_fullsized":"2023-11-24T06:20:22.642406+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:26.240404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653408+0000","last_change":"2023-11-24T06:20:22.655319+0000","last_active":"2023-11-24T06:20:23.653408+0000","last_peered":"2023-11-24T06:20:23.653408+0000","last_clean":"2023-11-24T06:20:23.653408+0000","last_became_active":"2023-11-24T06:20:22.654992+0000","last_became_peered":"2023-11-24T06:20:22.654992+0000","last_unstale":"2023-11-24T06:20:23.653408+0000","last_undegraded":"2023-11-24T06:20:23.653408+0000","last_fullsized":"2023-11-24T06:20:23.653408+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:09:36.482000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"52'8","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.664518+0000","last_change":"2023-11-24T06:20:21.634951+0000","last_active":"2023-11-24T06:20:22.664518+0000","last_peered":"2023-11-24T06:20:22.664518+0000","last_clean":"2023-11-24T06:20:22.664518+0000","last_became_active":"2023-11-24T06:20:21.634653+0000","last_became_peered":"2023-11-24T06:20:21.634653+0000","last_unstale":"2023-11-24T06:20:22.664518+0000","last_undegraded":"2023-11-24T06:20:22.664518+0000","last_fullsized":"2023-11-24T06:20:22.664518+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:34.255885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.654141+0000","last_change":"2023-11-24T06:20:22.641352+0000","last_active":"2023-11-24T06:20:23.654141+0000","last_peered":"2023-11-24T06:20:23.654141+0000","last_clean":"2023-11-24T06:20:23.654141+0000","last_became_active":"2023-11-24T06:20:22.641069+0000","last_became_peered":"2023-11-24T06:20:22.641069+0000","last_unstale":"2023-11-24T06:20:23.654141+0000","last_undegraded":"2023-11-24T06:20:23.654141+0000","last_fullsized":"2023-11-24T06:20:23.654141+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:17:17.159648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.661270+0000","last_change":"2023-11-24T06:20:21.633653+0000","last_active":"2023-11-24T06:20:22.661270+0000","last_peered":"2023-11-24T06:20:22.661270+0000","last_clean":"2023-11-24T06:20:22.661270+0000","last_became_active":"2023-11-24T06:20:21.633333+0000","last_became_peered":"2023-11-24T06:20:21.633333+0000","last_unstale":"2023-11-24T06:20:22.661270+0000","last_undegraded":"2023-11-24T06:20:22.661270+0000","last_fullsized":"2023-11-24T06:20:22.661270+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953077+0000","last_change":"2023-11-24T06:20:22.651717+0000","last_active":"2023-11-24T06:20:23.953077+0000","last_peered":"2023-11-24T06:20:23.953077+0000","last_clean":"2023-11-24T06:20:23.953077+0000","last_became_active":"2023-11-24T06:20:22.651384+0000","last_became_peered":"2023-11-24T06:20:22.651384+0000","last_unstale":"2023-11-24T06:20:23.953077+0000","last_undegraded":"2023-11-24T06:20:23.953077+0000","last_fullsized":"2023-11-24T06:20:23.953077+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:43:50.378588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.664283+0000","last_change":"2023-11-24T06:20:21.638281+0000","last_active":"2023-11-24T06:20:22.664283+0000","last_peered":"2023-11-24T06:20:22.664283+0000","last_clean":"2023-11-24T06:20:22.664283+0000","last_became_active":"2023-11-24T06:20:21.638014+0000","last_became_peered":"2023-11-24T06:20:21.638014+0000","last_unstale":"2023-11-24T06:20:22.664283+0000","last_undegraded":"2023-11-24T06:20:22.664283+0000","last_fullsized":"2023-11-24T06:20:22.664283+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:34.255885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952990+0000","last_change":"2023-11-24T06:20:22.648184+0000","last_active":"2023-11-24T06:20:23.952990+0000","last_peered":"2023-11-24T06:20:23.952990+0000","last_clean":"2023-11-24T06:20:23.952990+0000","last_became_active":"2023-11-24T06:20:22.647911+0000","last_became_peered":"2023-11-24T06:20:22.647911+0000","last_unstale":"2023-11-24T06:20:23.952990+0000","last_undegraded":"2023-11-24T06:20:23.952990+0000","last_fullsized":"2023-11-24T06:20:23.952990+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:08.604050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663097+0000","last_change":"2023-11-24T06:20:21.634745+0000","last_active":"2023-11-24T06:20:22.663097+0000","last_peered":"2023-11-24T06:20:22.663097+0000","last_clean":"2023-11-24T06:20:22.663097+0000","last_became_active":"2023-11-24T06:20:21.634453+0000","last_became_peered":"2023-11-24T06:20:21.634453+0000","last_unstale":"2023-11-24T06:20:22.663097+0000","last_undegraded":"2023-11-24T06:20:22.663097+0000","last_fullsized":"2023-11-24T06:20:22.663097+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:27:34.255885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653519+0000","last_change":"2023-11-24T06:20:22.638288+0000","last_active":"2023-11-24T06:20:23.653519+0000","last_peered":"2023-11-24T06:20:23.653519+0000","last_clean":"2023-11-24T06:20:23.653519+0000","last_became_active":"2023-11-24T06:20:22.638019+0000","last_became_peered":"2023-11-24T06:20:22.638019+0000","last_unstale":"2023-11-24T06:20:23.653519+0000","last_undegraded":"2023-11-24T06:20:23.653519+0000","last_fullsized":"2023-11-24T06:20:23.653519+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:32:55.130880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.640208+0000","last_change":"2023-11-24T06:20:21.640109+0000","last_active":"2023-11-24T06:20:22.640208+0000","last_peered":"2023-11-24T06:20:22.640208+0000","last_clean":"2023-11-24T06:20:22.640208+0000","last_became_active":"2023-11-24T06:20:21.639777+0000","last_became_peered":"2023-11-24T06:20:21.639777+0000","last_unstale":"2023-11-24T06:20:22.640208+0000","last_undegraded":"2023-11-24T06:20:22.640208+0000","last_fullsized":"2023-11-24T06:20:22.640208+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:30.394419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953139+0000","last_change":"2023-11-24T06:20:22.651732+0000","last_active":"2023-11-24T06:20:23.953139+0000","last_peered":"2023-11-24T06:20:23.953139+0000","last_clean":"2023-11-24T06:20:23.953139+0000","last_became_active":"2023-11-24T06:20:22.651227+0000","last_became_peered":"2023-11-24T06:20:22.651227+0000","last_unstale":"2023-11-24T06:20:23.953139+0000","last_undegraded":"2023-11-24T06:20:23.953139+0000","last_fullsized":"2023-11-24T06:20:23.953139+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:43:02.349069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.663397+0000","last_change":"2023-11-24T06:20:21.631857+0000","last_active":"2023-11-24T06:20:22.663397+0000","last_peered":"2023-11-24T06:20:22.663397+0000","last_clean":"2023-11-24T06:20:22.663397+0000","last_became_active":"2023-11-24T06:20:21.631396+0000","last_became_peered":"2023-11-24T06:20:21.631396+0000","last_unstale":"2023-11-24T06:20:22.663397+0000","last_undegraded":"2023-11-24T06:20:22.663397+0000","last_fullsized":"2023-11-24T06:20:22.663397+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:12:22.460606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952386+0000","last_change":"2023-11-24T06:20:22.648369+0000","last_active":"2023-11-24T06:20:23.952386+0000","last_peered":"2023-11-24T06:20:23.952386+0000","last_clean":"2023-11-24T06:20:23.952386+0000","last_became_active":"2023-11-24T06:20:22.647963+0000","last_became_peered":"2023-11-24T06:20:22.647963+0000","last_unstale":"2023-11-24T06:20:23.952386+0000","last_undegraded":"2023-11-24T06:20:23.952386+0000","last_fullsized":"2023-11-24T06:20:23.952386+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:30.081700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.634136+0000","last_change":"2023-11-24T06:20:21.633654+0000","last_active":"2023-11-24T06:20:22.634136+0000","last_peered":"2023-11-24T06:20:22.634136+0000","last_clean":"2023-11-24T06:20:22.634136+0000","last_became_active":"2023-11-24T06:20:21.633387+0000","last_became_peered":"2023-11-24T06:20:21.633387+0000","last_unstale":"2023-11-24T06:20:22.634136+0000","last_undegraded":"2023-11-24T06:20:22.634136+0000","last_fullsized":"2023-11-24T06:20:22.634136+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:24.432980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952097+0000","last_change":"2023-11-24T06:20:22.646935+0000","last_active":"2023-11-24T06:20:23.952097+0000","last_peered":"2023-11-24T06:20:23.952097+0000","last_clean":"2023-11-24T06:20:23.952097+0000","last_became_active":"2023-11-24T06:20:22.646663+0000","last_became_peered":"2023-11-24T06:20:22.646663+0000","last_unstale":"2023-11-24T06:20:23.952097+0000","last_undegraded":"2023-11-24T06:20:23.952097+0000","last_fullsized":"2023-11-24T06:20:23.952097+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:25:27.244946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.641390+0000","last_change":"2023-11-24T06:20:21.639872+0000","last_active":"2023-11-24T06:20:22.641390+0000","last_peered":"2023-11-24T06:20:22.641390+0000","last_clean":"2023-11-24T06:20:22.641390+0000","last_became_active":"2023-11-24T06:20:21.639600+0000","last_became_peered":"2023-11-24T06:20:21.639600+0000","last_unstale":"2023-11-24T06:20:22.641390+0000","last_undegraded":"2023-11-24T06:20:22.641390+0000","last_fullsized":"2023-11-24T06:20:22.641390+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:13:34.095847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.953092+0000","last_change":"2023-11-24T06:20:22.652562+0000","last_active":"2023-11-24T06:20:23.953092+0000","last_peered":"2023-11-24T06:20:23.953092+0000","last_clean":"2023-11-24T06:20:23.953092+0000","last_became_active":"2023-11-24T06:20:22.651592+0000","last_became_peered":"2023-11-24T06:20:22.651592+0000","last_unstale":"2023-11-24T06:20:23.953092+0000","last_undegraded":"2023-11-24T06:20:23.953092+0000","last_fullsized":"2023-11-24T06:20:23.953092+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:36:36.186702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.662133+0000","last_change":"2023-11-24T06:20:21.638221+0000","last_active":"2023-11-24T06:20:22.662133+0000","last_peered":"2023-11-24T06:20:22.662133+0000","last_clean":"2023-11-24T06:20:22.662133+0000","last_became_active":"2023-11-24T06:20:21.637800+0000","last_became_peered":"2023-11-24T06:20:21.637800+0000","last_unstale":"2023-11-24T06:20:22.662133+0000","last_undegraded":"2023-11-24T06:20:22.662133+0000","last_fullsized":"2023-11-24T06:20:22.662133+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:12.935014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952951+0000","last_change":"2023-11-24T06:20:22.648346+0000","last_active":"2023-11-24T06:20:23.952951+0000","last_peered":"2023-11-24T06:20:23.952951+0000","last_clean":"2023-11-24T06:20:23.952951+0000","last_became_active":"2023-11-24T06:20:22.646016+0000","last_became_peered":"2023-11-24T06:20:22.646016+0000","last_unstale":"2023-11-24T06:20:23.952951+0000","last_undegraded":"2023-11-24T06:20:23.952951+0000","last_fullsized":"2023-11-24T06:20:23.952951+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:48:30.081700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.641445+0000","last_change":"2023-11-24T06:20:21.635897+0000","last_active":"2023-11-24T06:20:22.641445+0000","last_peered":"2023-11-24T06:20:22.641445+0000","last_clean":"2023-11-24T06:20:22.641445+0000","last_became_active":"2023-11-24T06:20:21.635167+0000","last_became_peered":"2023-11-24T06:20:21.635167+0000","last_unstale":"2023-11-24T06:20:22.641445+0000","last_undegraded":"2023-11-24T06:20:22.641445+0000","last_fullsized":"2023-11-24T06:20:22.641445+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:13:34.095847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.652227+0000","last_change":"2023-11-24T06:20:22.644709+0000","last_active":"2023-11-24T06:20:23.652227+0000","last_peered":"2023-11-24T06:20:23.652227+0000","last_clean":"2023-11-24T06:20:23.652227+0000","last_became_active":"2023-11-24T06:20:22.644112+0000","last_became_peered":"2023-11-24T06:20:22.644112+0000","last_unstale":"2023-11-24T06:20:23.652227+0000","last_undegraded":"2023-11-24T06:20:23.652227+0000","last_fullsized":"2023-11-24T06:20:23.652227+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:40:31.398961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.641110+0000","last_change":"2023-11-24T06:20:21.633952+0000","last_active":"2023-11-24T06:20:22.641110+0000","last_peered":"2023-11-24T06:20:22.641110+0000","last_clean":"2023-11-24T06:20:22.641110+0000","last_became_active":"2023-11-24T06:20:21.633629+0000","last_became_peered":"2023-11-24T06:20:21.633629+0000","last_unstale":"2023-11-24T06:20:22.641110+0000","last_undegraded":"2023-11-24T06:20:22.641110+0000","last_fullsized":"2023-11-24T06:20:22.641110+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:17:23.296026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952770+0000","last_change":"2023-11-24T06:20:22.653035+0000","last_active":"2023-11-24T06:20:23.952770+0000","last_peered":"2023-11-24T06:20:23.952770+0000","last_clean":"2023-11-24T06:20:23.952770+0000","last_became_active":"2023-11-24T06:20:22.652698+0000","last_became_peered":"2023-11-24T06:20:22.652698+0000","last_unstale":"2023-11-24T06:20:23.952770+0000","last_undegraded":"2023-11-24T06:20:23.952770+0000","last_fullsized":"2023-11-24T06:20:23.952770+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:43:02.349069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.642278+0000","last_change":"2023-11-24T06:20:21.634883+0000","last_active":"2023-11-24T06:20:22.642278+0000","last_peered":"2023-11-24T06:20:22.642278+0000","last_clean":"2023-11-24T06:20:22.642278+0000","last_became_active":"2023-11-24T06:20:21.634624+0000","last_became_peered":"2023-11-24T06:20:21.634624+0000","last_unstale":"2023-11-24T06:20:22.642278+0000","last_undegraded":"2023-11-24T06:20:22.642278+0000","last_fullsized":"2023-11-24T06:20:22.642278+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:55:43.904787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952660+0000","last_change":"2023-11-24T06:20:22.656329+0000","last_active":"2023-11-24T06:20:23.952660+0000","last_peered":"2023-11-24T06:20:23.952660+0000","last_clean":"2023-11-24T06:20:23.952660+0000","last_became_active":"2023-11-24T06:20:22.652087+0000","last_became_peered":"2023-11-24T06:20:22.652087+0000","last_unstale":"2023-11-24T06:20:23.952660+0000","last_undegraded":"2023-11-24T06:20:23.952660+0000","last_fullsized":"2023-11-24T06:20:23.952660+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:38:40.237128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.635011+0000","last_change":"2023-11-24T06:20:21.635390+0000","last_active":"2023-11-24T06:20:22.635011+0000","last_peered":"2023-11-24T06:20:22.635011+0000","last_clean":"2023-11-24T06:20:22.635011+0000","last_became_active":"2023-11-24T06:20:21.635114+0000","last_became_peered":"2023-11-24T06:20:21.635114+0000","last_unstale":"2023-11-24T06:20:22.635011+0000","last_undegraded":"2023-11-24T06:20:22.635011+0000","last_fullsized":"2023-11-24T06:20:22.635011+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:18:34.137206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.653684+0000","last_change":"2023-11-24T06:20:22.646535+0000","last_active":"2023-11-24T06:20:23.653684+0000","last_peered":"2023-11-24T06:20:23.653684+0000","last_clean":"2023-11-24T06:20:23.653684+0000","last_became_active":"2023-11-24T06:20:22.646260+0000","last_became_peered":"2023-11-24T06:20:22.646260+0000","last_unstale":"2023-11-24T06:20:23.653684+0000","last_undegraded":"2023-11-24T06:20:23.653684+0000","last_fullsized":"2023-11-24T06:20:23.653684+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:41.454704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.633444+0000","last_change":"2023-11-24T06:20:21.636357+0000","last_active":"2023-11-24T06:20:22.633444+0000","last_peered":"2023-11-24T06:20:22.633444+0000","last_clean":"2023-11-24T06:20:22.633444+0000","last_became_active":"2023-11-24T06:20:21.636068+0000","last_became_peered":"2023-11-24T06:20:21.636068+0000","last_unstale":"2023-11-24T06:20:22.633444+0000","last_undegraded":"2023-11-24T06:20:22.633444+0000","last_fullsized":"2023-11-24T06:20:22.633444+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:59:38.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.651753+0000","last_change":"2023-11-24T06:20:22.646637+0000","last_active":"2023-11-24T06:20:23.651753+0000","last_peered":"2023-11-24T06:20:23.651753+0000","last_clean":"2023-11-24T06:20:23.651753+0000","last_became_active":"2023-11-24T06:20:22.646370+0000","last_became_peered":"2023-11-24T06:20:22.646370+0000","last_unstale":"2023-11-24T06:20:23.651753+0000","last_undegraded":"2023-11-24T06:20:23.651753+0000","last_fullsized":"2023-11-24T06:20:23.651753+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:16.523717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.634140+0000","last_change":"2023-11-24T06:20:21.641031+0000","last_active":"2023-11-24T06:20:22.634140+0000","last_peered":"2023-11-24T06:20:22.634140+0000","last_clean":"2023-11-24T06:20:22.634140+0000","last_became_active":"2023-11-24T06:20:21.640826+0000","last_became_peered":"2023-11-24T06:20:21.640826+0000","last_unstale":"2023-11-24T06:20:22.634140+0000","last_undegraded":"2023-11-24T06:20:22.634140+0000","last_fullsized":"2023-11-24T06:20:22.634140+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:40:25.850351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952522+0000","last_change":"2023-11-24T06:20:22.651781+0000","last_active":"2023-11-24T06:20:23.952522+0000","last_peered":"2023-11-24T06:20:23.952522+0000","last_clean":"2023-11-24T06:20:23.952522+0000","last_became_active":"2023-11-24T06:20:22.651361+0000","last_became_peered":"2023-11-24T06:20:22.651361+0000","last_unstale":"2023-11-24T06:20:23.952522+0000","last_undegraded":"2023-11-24T06:20:23.952522+0000","last_fullsized":"2023-11-24T06:20:23.952522+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:15:09.954871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.640147+0000","last_change":"2023-11-24T06:20:21.636646+0000","last_active":"2023-11-24T06:20:22.640147+0000","last_peered":"2023-11-24T06:20:22.640147+0000","last_clean":"2023-11-24T06:20:22.640147+0000","last_became_active":"2023-11-24T06:20:21.636335+0000","last_became_peered":"2023-11-24T06:20:21.636335+0000","last_unstale":"2023-11-24T06:20:22.640147+0000","last_undegraded":"2023-11-24T06:20:22.640147+0000","last_fullsized":"2023-11-24T06:20:22.640147+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:31:53.216779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-24T06:20:23.952434+0000","last_change":"2023-11-24T06:20:22.653597+0000","last_active":"2023-11-24T06:20:23.952434+0000","last_peered":"2023-11-24T06:20:23.952434+0000","last_clean":"2023-11-24T06:20:23.952434+0000","last_became_active":"2023-11-24T06:20:22.651007+0000","last_became_peered":"2023-11-24T06:20:22.651007+0000","last_unstale":"2023-11-24T06:20:23.952434+0000","last_undegraded":"2023-11-24T06:20:23.952434+0000","last_fullsized":"2023-11-24T06:20:23.952434+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:21.614935+0000","last_clean_scrub_stamp":"2023-11-24T06:20:21.614935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:03:16.942755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.638921+0000","last_change":"2023-11-24T06:20:21.635744+0000","last_active":"2023-11-24T06:20:22.638921+0000","last_peered":"2023-11-24T06:20:22.638921+0000","last_clean":"2023-11-24T06:20:22.638921+0000","last_became_active":"2023-11-24T06:20:21.635447+0000","last_became_peered":"2023-11-24T06:20:21.635447+0000","last_unstale":"2023-11-24T06:20:22.638921+0000","last_undegraded":"2023-11-24T06:20:22.638921+0000","last_fullsized":"2023-11-24T06:20:22.638921+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:20.610393+0000","last_clean_scrub_stamp":"2023-11-24T06:20:20.610393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:42:21.256791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":197,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-24T06:20:22.633273+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:22.633273+0000","last_peered":"2023-11-24T06:20:22.633273+0000","last_clean":"2023-11-24T06:20:22.633273+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:22.633273+0000","last_undegraded":"2023-11-24T06:20:22.633273+0000","last_fullsized":"2023-11-24T06:20:22.633273+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28228,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343612,"statfs":{"total":96636764160,"available":96607858688,"internally_reserved":0,"allocated":1236992,"data_stored":860946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.72299999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60599999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.65300000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.71199999999999997}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.504},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.623},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67100000000000004}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.755},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.63800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":270418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":6.423,"5min":6.423,"15min":6.423},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.4699999999999998,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.505}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.46899999999999997}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":12.554,"5min":12.554,"15min":12.554},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.625},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.65500000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":6.1280000000000001,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":270508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.311}]},{"osd":3,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.29199999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.62},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:19:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":862980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.441}]},{"osd":3,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.42399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.77200000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.72099999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.88900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":270474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.308},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.374}]},{"osd":4,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.753}]},{"osd":5,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.71399999999999997}]},{"osd":6,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.82599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":270474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.27000000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.40000000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.58799999999999997}]},{"osd":5,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.77000000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.621}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":272030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.44500000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.751}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":271070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.04,"5min":3.04,"15min":3.04},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.621},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.1319999999999997,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.501}]},{"osd":3,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":10.198,"5min":10.198,"15min":10.198},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":10.215999999999999,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.66400000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.70099999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.64800000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:20:31.680 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T06:20:31.680 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T06:20:31.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.839+0000 7f74e3492700 1 Processor -- start 2023-11-24T06:20:31.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.839+0000 7f74e3492700 1 -- start start 2023-11-24T06:20:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.840+0000 7f74e3492700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f74dc0d05b0 0x7f74dc0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.840+0000 7f74e3492700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0b2a60 con 0x7f74dc06e930 2023-11-24T06:20:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.840+0000 7f74e3492700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc15e970 con 0x7f74dc15ac40 2023-11-24T06:20:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.840+0000 7f74e3492700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74dc15ef30 con 0x7f74dc0d05b0 2023-11-24T06:20:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.840+0000 7f74e0a2d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f74dc0d05b0 0x7f74dc0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.840+0000 7f74e0a2d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f74dc0d05b0 0x7f74dc0b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38062/0 (socket says 172.21.15.89:38062) 2023-11-24T06:20:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.840+0000 7f74e0a2d700 1 -- 172.21.15.89:0/1362864430 learned_addr learned my addr 172.21.15.89:0/1362864430 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.840+0000 7f74e1a2f700 1 --1- 172.21.15.89:0/1362864430 >> v1:172.21.15.89:6789/0 conn(0x7f74dc06e930 0x7f74dc0ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41568/0 (socket says 172.21.15.89:41568) 2023-11-24T06:20:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.841+0000 7f74e0a2d700 1 -- 172.21.15.89:0/1362864430 >> v1:172.21.15.123:6789/0 conn(0x7f74dc15ac40 legacy=0x7f74dc0cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.841+0000 7f74e0a2d700 1 -- 172.21.15.89:0/1362864430 >> v1:172.21.15.89:6789/0 conn(0x7f74dc06e930 legacy=0x7f74dc0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.841+0000 7f74e0a2d700 1 -- 172.21.15.89:0/1362864430 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc0b2cb0 con 0x7f74dc0d05b0 2023-11-24T06:20:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.841+0000 7f74e0a2d700 1 --2- 172.21.15.89:0/1362864430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc0d05b0 0x7f74dc0b42d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f74d80098f0 tx=0x7f74d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd62bb9ec48f686a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.841+0000 7f74d3fff700 1 -- 172.21.15.89:0/1362864430 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d800b8a0 con 0x7f74dc0d05b0 2023-11-24T06:20:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.841+0000 7f74d3fff700 1 -- 172.21.15.89:0/1362864430 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d800ba00 con 0x7f74dc0d05b0 2023-11-24T06:20:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.841+0000 7f74d3fff700 1 -- 172.21.15.89:0/1362864430 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d800bd90 con 0x7f74dc0d05b0 2023-11-24T06:20:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.842+0000 7f74e3492700 1 -- 172.21.15.89:0/1362864430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc0d05b0 msgr2=0x7f74dc0b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.842+0000 7f74e3492700 1 --2- 172.21.15.89:0/1362864430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc0d05b0 0x7f74dc0b42d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f74d80098f0 tx=0x7f74d8009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.842+0000 7f74e3492700 1 -- 172.21.15.89:0/1362864430 shutdown_connections 2023-11-24T06:20:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.842+0000 7f74e3492700 1 --2- 172.21.15.89:0/1362864430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc0d05b0 0x7f74dc0b42d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.842+0000 7f74e3492700 1 -- 172.21.15.89:0/1362864430 >> 172.21.15.89:0/1362864430 conn(0x7f74dc06b410 msgr2=0x7f74dc06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.842+0000 7f74e3492700 1 -- 172.21.15.89:0/1362864430 shutdown_connections 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.842+0000 7f74e3492700 1 -- 172.21.15.89:0/1362864430 wait complete. 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.843+0000 7f74e3492700 1 Processor -- start 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.843+0000 7f74e3492700 1 -- start start 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e3492700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc06e930 0x7f74dc0ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e3492700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74dc0d05b0 0x7f74dc14f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e3492700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74dc15ac40 0x7f74dc0c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e3492700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0caac0 con 0x7f74dc06e930 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e3492700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0cac50 con 0x7f74dc15ac40 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e3492700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc14fb10 con 0x7f74dc0d05b0 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc06e930 0x7f74dc0ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc06e930 0x7f74dc0ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38066/0 (socket says 172.21.15.89:38066) 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 -- 172.21.15.89:0/2490706937 learned_addr learned my addr 172.21.15.89:0/2490706937 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e0a2d700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74dc0d05b0 0x7f74dc14f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 -- 172.21.15.89:0/2490706937 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74dc0d05b0 msgr2=0x7f74dc14f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e1a2f700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74dc15ac40 0x7f74dc0c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74dc0d05b0 0x7f74dc14f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 -- 172.21.15.89:0/2490706937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74dc15ac40 msgr2=0x7f74dc0c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74dc15ac40 0x7f74dc0c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 -- 172.21.15.89:0/2490706937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d8009550 con 0x7f74dc06e930 2023-11-24T06:20:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e122e700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc06e930 0x7f74dc0ad870 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c4e0 tx=0x7f74cc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74d1ffb700 1 -- 172.21.15.89:0/2490706937 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc0052d0 con 0x7f74dc06e930 2023-11-24T06:20:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74d1ffb700 1 -- 172.21.15.89:0/2490706937 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc005430 con 0x7f74dc06e930 2023-11-24T06:20:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74d1ffb700 1 -- 172.21.15.89:0/2490706937 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc0038c0 con 0x7f74dc06e930 2023-11-24T06:20:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.844+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc0ade50 con 0x7f74dc06e930 2023-11-24T06:20:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.845+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc0ae410 con 0x7f74dc06e930 2023-11-24T06:20:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.845+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc068480 con 0x7f74dc06e930 2023-11-24T06:20:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.846+0000 7f74d1ffb700 1 -- 172.21.15.89:0/2490706937 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f74cc003a20 con 0x7f74dc06e930 2023-11-24T06:20:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.846+0000 7f74d1ffb700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f74c006aed0 0x7f74c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.846+0000 7f74d1ffb700 1 -- 172.21.15.89:0/2490706937 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74cc088ac0 con 0x7f74dc06e930 2023-11-24T06:20:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.846+0000 7f74e0a2d700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f74c006aed0 0x7f74c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.847+0000 7f74e0a2d700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f74c006aed0 0x7f74c006d390 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0b16c0 tx=0x7f74d8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:31.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:31.850+0000 7f74d1ffb700 1 -- 172.21.15.89:0/2490706937 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f74cc053b60 con 0x7f74dc06e930 2023-11-24T06:20:32.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.121+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f74dc068fb0 con 0x7f74dc06e930 2023-11-24T06:20:32.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.123+0000 7f74d1ffb700 1 -- 172.21.15.89:0/2490706937 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f74cc0170e0 con 0x7f74dc06e930 2023-11-24T06:20:32.125 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2023-11-24T06:20:32.126 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlwIfyKhAAMZcwOQGAm+XcH+mUKcTiUg== 2023-11-24T06:20:32.126 INFO:teuthology.orchestra.run.smithi089.stdout: caps mds = "allow" 2023-11-24T06:20:32.126 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "allow r" 2023-11-24T06:20:32.126 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T06:20:32.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.125+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f74c006aed0 msgr2=0x7f74c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:32.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.125+0000 7f74e3492700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f74c006aed0 0x7f74c006d390 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0b16c0 tx=0x7f74d8008030 comp rx=0 tx=0).stop 2023-11-24T06:20:32.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc06e930 msgr2=0x7f74dc0ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:32.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc06e930 0x7f74dc0ad870 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c4e0 tx=0x7f74cc00c8a0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 shutdown_connections 2023-11-24T06:20:32.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f74c006aed0 0x7f74c006d390 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74dc15ac40 0x7f74dc0c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74dc06e930 0x7f74dc0ad870 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 --2- 172.21.15.89:0/2490706937 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74dc0d05b0 0x7f74dc14f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 >> 172.21.15.89:0/2490706937 conn(0x7f74dc06b410 msgr2=0x7f74dc159d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:32.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 shutdown_connections 2023-11-24T06:20:32.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.126+0000 7f74e3492700 1 -- 172.21.15.89:0/2490706937 wait complete. 2023-11-24T06:20:32.130 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2023-11-24T06:20:32.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:32.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.257+0000 7f69af1ca700 1 Processor -- start 2023-11-24T06:20:32.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.257+0000 7f69af1ca700 1 -- start start 2023-11-24T06:20:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69af1ca700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f69a80cf360 0x7f69a815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69af1ca700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a806cf40 0x7f69a80b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69af1ca700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a80b4c40 con 0x7f69a806cac0 2023-11-24T06:20:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69af1ca700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a80cf8b0 con 0x7f69a80cf360 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69af1ca700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a8070c60 con 0x7f69a806cf40 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69a7fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f69a806cac0 0x7f69a80b3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41580/0 (socket says 172.21.15.89:41580) 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69a7fff700 1 -- 172.21.15.89:0/2071737371 learned_addr learned my addr 172.21.15.89:0/2071737371 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69acf66700 1 --2- 172.21.15.89:0/2071737371 >> v2:172.21.15.123:3300/0 conn(0x7f69a80cf360 0x7f69a815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369393946 0 0) 0x7f69a80b4c40 con 0x7f69a806cac0 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69980035b0 con 0x7f69a806cac0 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.258+0000 7f69ad767700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a806cf40 0x7f69a80b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1355553867 0 0) 0x7f69980035b0 con 0x7f69a806cac0 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a806cf40 msgr2=0x7f69a80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a806cf40 0x7f69a80b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 >> v2:172.21.15.123:3300/0 conn(0x7f69a80cf360 msgr2=0x7f69a815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 --2- 172.21.15.89:0/2071737371 >> v2:172.21.15.123:3300/0 conn(0x7f69a80cf360 0x7f69a815a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8070f00 con 0x7f69a806cac0 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f6994001670 con 0x7f69a806cac0 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 >> v1:172.21.15.89:6789/0 conn(0x7f69a806cac0 legacy=0x7f69a80b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a806cf40 0x7f6998003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6998003f80 0x7f6998004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f6998005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80cf8b0 con 0x7f6998004d90 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a8070c60 con 0x7f6998003f80 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699800ba10 con 0x7f69a806cf40 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69ad767700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f6998005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69acf66700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a806cf40 0x7f6998003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a7fff700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6998003f80 0x7f6998004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69ad767700 1 -- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a806cf40 msgr2=0x7f6998003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69ad767700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a806cf40 0x7f6998003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69ad767700 1 -- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6998003f80 msgr2=0x7f6998004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69ad767700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6998003f80 0x7f6998004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69ad767700 1 -- 172.21.15.89:0/2071737371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69980035b0 con 0x7f6998004d90 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69ad767700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f6998005210 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f69a0009f90 tx=0x7f6998005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1cadf4314e78c1c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a000ebd0 con 0x7f6998004d90 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a000f040 con 0x7f6998004d90 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.259+0000 7f69a77fe700 1 -- 172.21.15.89:0/2071737371 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0014480 con 0x7f6998004d90 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.260+0000 7f69af1ca700 1 -- 172.21.15.89:0/2071737371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 msgr2=0x7f6998005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.260+0000 7f69af1ca700 1 --2- 172.21.15.89:0/2071737371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f6998005210 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f69a0009f90 tx=0x7f6998005880 comp rx=0 tx=0).stop 2023-11-24T06:20:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.260+0000 7f69ad767700 1 -- 172.21.15.89:0/2071737371 reap_dead start 2023-11-24T06:20:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.260+0000 7f69af1ca700 1 -- 172.21.15.89:0/2071737371 shutdown_connections 2023-11-24T06:20:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.260+0000 7f69af1ca700 1 -- 172.21.15.89:0/2071737371 >> 172.21.15.89:0/2071737371 conn(0x7f69a80b1740 msgr2=0x7f69a80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.260+0000 7f69af1ca700 1 -- 172.21.15.89:0/2071737371 shutdown_connections 2023-11-24T06:20:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.260+0000 7f69af1ca700 1 -- 172.21.15.89:0/2071737371 wait complete. 2023-11-24T06:20:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.261+0000 7f69af1ca700 1 Processor -- start 2023-11-24T06:20:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.262+0000 7f69af1ca700 1 -- start start 2023-11-24T06:20:32.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69af1ca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f69a814f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69af1ca700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a814f060 0x7f69a814e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69af1ca700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a80cd760 0x7f69a80cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69af1ca700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80ce130 con 0x7f6998004d90 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69af1ca700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a814ee80 con 0x7f69a814f060 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69af1ca700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815afa0 con 0x7f69a80cd760 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f69a814f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f69a814f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38086/0 (socket says 172.21.15.89:38086) 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 -- 172.21.15.89:0/347662282 learned_addr learned my addr 172.21.15.89:0/347662282 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69a7fff700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a814f060 0x7f69a814e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69ad767700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a80cd760 0x7f69a80cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 -- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a80cd760 msgr2=0x7f69a80cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:32.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a80cd760 0x7f69a80cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 -- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a814f060 msgr2=0x7f69a814e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a814f060 0x7f69a814e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 -- 172.21.15.89:0/347662282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a815acc0 con 0x7f6998004d90 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69ad767700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a80cd760 0x7f69a80cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69acf66700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f69a814f590 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f699c011aa0 tx=0x7f699c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69a57fa700 1 -- 172.21.15.89:0/347662282 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c00f9e0 con 0x7f6998004d90 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69a57fa700 1 -- 172.21.15.89:0/347662282 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c00fb40 con 0x7f6998004d90 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69a57fa700 1 -- 172.21.15.89:0/347662282 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c014440 con 0x7f6998004d90 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a815b400 con 0x7f6998004d90 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.263+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a815b9c0 con 0x7f6998004d90 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.264+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a81578a0 con 0x7f6998004d90 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.265+0000 7f69a57fa700 1 -- 172.21.15.89:0/347662282 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f699c0145a0 con 0x7f6998004d90 2023-11-24T06:20:32.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.265+0000 7f69a57fa700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f698c06aed0 0x7f698c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:32.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.265+0000 7f69a57fa700 1 -- 172.21.15.89:0/347662282 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f699c089da0 con 0x7f6998004d90 2023-11-24T06:20:32.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.265+0000 7f69a7fff700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f698c06aed0 0x7f698c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:32.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.265+0000 7f69a7fff700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f698c06aed0 0x7f698c06d390 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6994000fb0 tx=0x7f6994009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:32.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.270+0000 7f69a57fa700 1 -- 172.21.15.89:0/347662282 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f699c054e50 con 0x7f6998004d90 2023-11-24T06:20:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.505+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69a80751e0 con 0x7f6998004d90 2023-11-24T06:20:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.505+0000 7f69a57fa700 1 -- 172.21.15.89:0/347662282 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f699c058470 con 0x7f6998004d90 2023-11-24T06:20:32.509 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:32.509 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2011089843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2011089843},{"type":"v1","addr":"172.21.15.89:6836","nonce":2011089843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4955,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/2182962276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2182962276},{"type":"v1","addr":"172.21.15.123:6834","nonce":2182962276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2982219407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2982219407},{"type":"v1","addr":"172.21.15.123:6835","nonce":2982219407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:22.026873+0000","modified":"2023-11-24T06:20:25.053779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5577":{"gid":5577,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1341821445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1341821445},{"type":"v1","addr":"172.21.15.89:6837","nonce":1341821445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T06:20:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f698c06aed0 msgr2=0x7f698c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f698c06aed0 0x7f698c06d390 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6994000fb0 tx=0x7f6994009040 comp rx=0 tx=0).stop 2023-11-24T06:20:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 msgr2=0x7f69a814f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f69a814f590 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f699c011aa0 tx=0x7f699c011e60 comp rx=0 tx=0).stop 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 shutdown_connections 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69a80cd760 0x7f69a80cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6998004d90 0x7f69a814f590 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f698c06aed0 0x7f698c06d390 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 --2- 172.21.15.89:0/347662282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a814f060 0x7f69a814e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 >> 172.21.15.89:0/347662282 conn(0x7f69a80b1740 msgr2=0x7f69a806e600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 shutdown_connections 2023-11-24T06:20:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:32.507+0000 7f69af1ca700 1 -- 172.21.15.89:0/347662282 wait complete. 2023-11-24T06:20:32.515 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 31 2023-11-24T06:20:32.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 31, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T06:20:22.026873+0000', 'modified': '2023-11-24T06:20:25.053779+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5577}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5577': {'gid': 5577, 'name': 'c', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/1341821445', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6835', 'nonce': 1341821445}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 1341821445}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T06:20:32.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T06:20:32.925 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-24T06:20:33.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.047+0000 7f9504c65700 1 Processor -- start 2023-11-24T06:20:33.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.047+0000 7f9504c65700 1 -- start start 2023-11-24T06:20:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f9504c65700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95000adf00 0x7f950006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f9504c65700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f950006c540 0x7f95000b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f9504c65700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95000af950 con 0x7f95001446a0 2023-11-24T06:20:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f9504c65700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9500144a80 con 0x7f950006c540 2023-11-24T06:20:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f9504c65700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950006c9b0 con 0x7f95000adf00 2023-11-24T06:20:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fe59c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f95001446a0 0x7f95000b5ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41598/0 (socket says 172.21.15.89:41598) 2023-11-24T06:20:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fe59c700 1 -- 172.21.15.89:0/700909722 learned_addr learned my addr 172.21.15.89:0/700909722 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fed9d700 1 --2- 172.21.15.89:0/700909722 >> v2:172.21.15.123:3300/0 conn(0x7f950006c540 0x7f95000b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468693660 0 0) 0x7f95000af950 con 0x7f95001446a0 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95000adf00 0x7f950006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e80035b0 con 0x7f95001446a0 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3818279090 0 0) 0x7f94e80035b0 con 0x7f95001446a0 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95000adf00 msgr2=0x7f950006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95000adf00 0x7f950006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 >> v2:172.21.15.123:3300/0 conn(0x7f950006c540 msgr2=0x7f95000b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 --2- 172.21.15.89:0/700909722 >> v2:172.21.15.123:3300/0 conn(0x7f950006c540 0x7f95000b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95000afc30 con 0x7f95001446a0 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f94f4001670 con 0x7f95001446a0 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 >> v1:172.21.15.89:6789/0 conn(0x7f95001446a0 legacy=0x7f95000b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95000adf00 0x7f94e8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f94e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94e8004d90 0x7f94e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9500144a80 con 0x7f94e8003fb0 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950006c9b0 con 0x7f94e8004d90 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e800ba10 con 0x7f95000adf00 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.049+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f94e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fe59c700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95000adf00 0x7f94e8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fed9d700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94e8004d90 0x7f94e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fdd9b700 1 -- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95000adf00 msgr2=0x7f94e8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95000adf00 0x7f94e8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fdd9b700 1 -- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94e8004d90 msgr2=0x7f94e8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94e8004d90 0x7f94e8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fdd9b700 1 -- 172.21.15.89:0/700909722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e80035b0 con 0x7f94e8003fb0 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fed9d700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94e8004d90 0x7f94e8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f94e8004850 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009b30 tx=0x7f94e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ccb2f974c57cb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f000e6c0 con 0x7f94e8003fb0 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f000e820 con 0x7f94e8003fb0 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f94fd59a700 1 -- 172.21.15.89:0/700909722 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f000eb90 con 0x7f94e8003fb0 2023-11-24T06:20:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f9504c65700 1 -- 172.21.15.89:0/700909722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 msgr2=0x7f94e8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.050+0000 7f9504c65700 1 --2- 172.21.15.89:0/700909722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f94e8004850 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009b30 tx=0x7f94e8005880 comp rx=0 tx=0).stop 2023-11-24T06:20:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.051+0000 7f94fed9d700 1 -- 172.21.15.89:0/700909722 reap_dead start 2023-11-24T06:20:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.051+0000 7f9504c65700 1 -- 172.21.15.89:0/700909722 shutdown_connections 2023-11-24T06:20:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.051+0000 7f9504c65700 1 -- 172.21.15.89:0/700909722 >> 172.21.15.89:0/700909722 conn(0x7f9500071b90 msgr2=0x7f9500071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.051+0000 7f9504c65700 1 -- 172.21.15.89:0/700909722 shutdown_connections 2023-11-24T06:20:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.051+0000 7f9504c65700 1 -- 172.21.15.89:0/700909722 wait complete. 2023-11-24T06:20:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.052+0000 7f9504c65700 1 Processor -- start 2023-11-24T06:20:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.052+0000 7f9504c65700 1 -- start start 2023-11-24T06:20:33.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f9504c65700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f95001526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f9504c65700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9500142be0 0x7f950015d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f9504c65700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950015dea0 0x7f9500136500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f9504c65700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9500142420 con 0x7f94e8003fb0 2023-11-24T06:20:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f9504c65700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9500152ed0 con 0x7f950015dea0 2023-11-24T06:20:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f9504c65700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95001531b0 con 0x7f9500142be0 2023-11-24T06:20:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f95001526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f95001526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38106/0 (socket says 172.21.15.89:38106) 2023-11-24T06:20:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 -- 172.21.15.89:0/1595508764 learned_addr learned my addr 172.21.15.89:0/1595508764 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9500142be0 0x7f950015d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fed9d700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950015dea0 0x7f9500136500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 -- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9500142be0 msgr2=0x7f950015d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9500142be0 0x7f950015d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 -- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950015dea0 msgr2=0x7f9500136500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950015dea0 0x7f9500136500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 -- 172.21.15.89:0/1595508764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95001426c0 con 0x7f94e8003fb0 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fed9d700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950015dea0 0x7f9500136500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94fe59c700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f95001526a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f94f40072c0 tx=0x7f94f4007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94eeffd700 1 -- 172.21.15.89:0/1595508764 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f400f870 con 0x7f94e8003fb0 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94eeffd700 1 -- 172.21.15.89:0/1595508764 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f400f9f0 con 0x7f94e8003fb0 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f94eeffd700 1 -- 172.21.15.89:0/1595508764 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f400fda0 con 0x7f94e8003fb0 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.053+0000 7f9504c65700 1 -- 172.21.15.89:0/1595508764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9500152be0 con 0x7f94e8003fb0 2023-11-24T06:20:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.054+0000 7f9504c65700 1 -- 172.21.15.89:0/1595508764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9500143060 con 0x7f94e8003fb0 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.055+0000 7f94eeffd700 1 -- 172.21.15.89:0/1595508764 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f94f401d460 con 0x7f94e8003fb0 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.055+0000 7f94eeffd700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94e006aed0 0x7f94e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.055+0000 7f94eeffd700 1 -- 172.21.15.89:0/1595508764 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94f4089600 con 0x7f94e8003fb0 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.055+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94e006aed0 0x7f94e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.056+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94e006aed0 0x7f94e006d390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f94f00090b0 tx=0x7f94f001e040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.057+0000 7f9510e5eb80 1 Processor -- start 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.058+0000 7f9510e5eb80 1 -- start start 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.058+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571379840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.058+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571348ed0 0x563571379d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.058+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x56357135ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.058+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635712a6770 con 0x563571348ed0 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.058+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357128d750 con 0x563571330490 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.058+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x56357128b070 con 0x563571333080 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571348ed0 0x563571379d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571348ed0 0x563571379d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38114/0 (socket says 172.21.15.89:38114) 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 -- 172.21.15.89:0/2704546735 learned_addr learned my addr 172.21.15.89:0/2704546735 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fe59c700 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571379840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fed9d700 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x56357135ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 -- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 msgr2=0x563571379840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571379840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 -- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 msgr2=0x56357135ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x56357135ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 -- 172.21.15.89:0/2704546735 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635711035e0 con 0x563571348ed0 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fe59c700 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571379840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571348ed0 0x563571379d80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f94f000e4b0 tx=0x7f94f0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3089204635b1ad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94dffff700 1 -- 172.21.15.89:0/2704546735 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f002d070 con 0x563571348ed0 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94dffff700 1 -- 172.21.15.89:0/2704546735 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f000ee60 con 0x563571348ed0 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.059+0000 7f94dffff700 1 -- 172.21.15.89:0/2704546735 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0021040 con 0x563571348ed0 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 -- 172.21.15.89:0/2704546735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571348ed0 msgr2=0x563571379d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571348ed0 0x563571379d80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f94f000e4b0 tx=0x7f94f0013000 comp rx=0 tx=0).stop 2023-11-24T06:20:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 -- 172.21.15.89:0/2704546735 shutdown_connections 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x56357135ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571379840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/2704546735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571348ed0 0x563571379d80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 -- 172.21.15.89:0/2704546735 >> 172.21.15.89:0/2704546735 conn(0x563571261310 msgr2=0x563571377e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 -- 172.21.15.89:0/2704546735 shutdown_connections 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.060+0000 7f9510e5eb80 1 -- 172.21.15.89:0/2704546735 wait complete. 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.061+0000 7f9510e5eb80 1 -- start start 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.061+0000 7f9510e5eb80 1 Processor -- start 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.061+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x5635713725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571333080 0x56357135ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571348ed0 0x563571338cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563571286960 con 0x563571333080 2023-11-24T06:20:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x563571240510 con 0x563571330490 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635712401b0 con 0x563571348ed0 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 10 client.0 _ll_register_callbacks cb 0x56357116ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571333080 0x56357135ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571333080 0x56357135ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38122/0 (socket says 172.21.15.89:38122) 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 -- 172.21.15.89:0/3869685851 learned_addr learned my addr 172.21.15.89:0/3869685851 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fe59c700 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x5635713725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fed9d700 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571348ed0 0x563571338cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 -- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571348ed0 msgr2=0x563571338cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571348ed0 0x563571338cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 -- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 msgr2=0x5635713725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x5635713725e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 -- 172.21.15.89:0/3869685851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e80035b0 con 0x563571333080 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fe59c700 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x5635713725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571333080 0x56357135ce40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f94f000ec70 tx=0x7f94f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf6d50b84623c00 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94dd7fa700 1 -- 172.21.15.89:0/3869685851 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0021040 con 0x563571333080 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94dd7fa700 1 -- 172.21.15.89:0/3869685851 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94f001c650 con 0x563571333080 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3869685851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563571325a40 con 0x563571333080 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94dd7fa700 1 -- 172.21.15.89:0/3869685851 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94f001c810 con 0x563571333080 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 10 client.5751 fetch_fsmap learned FSMap version 31 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3869685851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x5635710afd90 con 0x563571333080 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.062+0000 7f94dd7fa700 1 -- 172.21.15.89:0/3869685851 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563571325a40 con 0x563571333080 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f94dd7fa700 1 -- 172.21.15.89:0/3869685851 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 31) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f94f0038030 con 0x563571333080 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f9510e5eb80 10 client.5751 fetch_fsmap finished waiting for FSMap version 31 2023-11-24T06:20:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f9510e5eb80 10 client.5751 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f9510e5eb80 20 client.5751 populate_metadata read hostname 'smithi089' 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] conn(0x563571371ed0 0x5635713601e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f9510e5eb80 4 client.5751 mds_command: new command op to 5577 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3869685851 --> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563571301840 con 0x563571371ed0 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f94fe59c700 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] conn(0x563571371ed0 0x5635713601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f94fe59c700 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] conn(0x563571371ed0 0x5635713601e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b54708dc40142c60 server_cookie=b285884fef5d1b6f in_seq=0 out_seq=0 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.063+0000 7f94dd7fa700 10 client.5751 ms_handle_connect on v2:172.21.15.89:6835/1341821445 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f94dd7fa700 1 -- 172.21.15.89:0/3869685851 <== mds.0 v2:172.21.15.89:6835/1341821445 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563571301840 con 0x563571371ed0 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f94dd7fa700 10 client.5751 handle_command_reply: tid=0 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 client.5751 shutdown 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3869685851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571333080 msgr2=0x56357135ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571333080 0x56357135ce40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f94f000ec70 tx=0x7f94f0004750 comp rx=0 tx=0).stop 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3869685851 shutdown_connections 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571348ed0 0x563571338cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571330490 0x5635713725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] conn(0x563571371ed0 0x5635713601e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3869685851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571333080 0x56357135ce40 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f94dd7fa700 0 client.5751 ms_handle_reset on v2:172.21.15.89:6835/1341821445 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3869685851 >> 172.21.15.89:0/3869685851 conn(0x563571261310 msgr2=0x56357137bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3869685851 shutdown_connections 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3869685851 wait complete. 2023-11-24T06:20:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.064+0000 7f9510e5eb80 20 client.5751 trim_cache size 0 max 16384 2023-11-24T06:20:33.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.082+0000 7f9510e5eb80 1 Processor -- start 2023-11-24T06:20:33.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.082+0000 7f9510e5eb80 1 -- start start 2023-11-24T06:20:33.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x563571326f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x56357132e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571374510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635711058d0 con 0x563571330490 2023-11-24T06:20:33.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357132dff0 con 0x563571348ed0 2023-11-24T06:20:33.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x56357130f340 con 0x563571333080 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f94fe59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x563571326f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f94fe59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x563571326f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38128/0 (socket says 172.21.15.89:38128) 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.083+0000 7f94fe59c700 1 -- 172.21.15.89:0/3781414678 learned_addr learned my addr 172.21.15.89:0/3781414678 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94fe59c700 1 -- 172.21.15.89:0/3781414678 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 msgr2=0x56357132e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94fe59c700 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x56357132e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94fe59c700 1 -- 172.21.15.89:0/3781414678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 msgr2=0x563571374510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94fed9d700 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571374510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94fe59c700 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571374510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94fe59c700 1 -- 172.21.15.89:0/3781414678 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563571098c10 con 0x563571330490 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94fed9d700 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571374510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94fe59c700 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x563571326f50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f94f4087b60 tx=0x7f94f4087a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8025b81858201cd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94ecff9700 1 -- 172.21.15.89:0/3781414678 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4086070 con 0x563571330490 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94ecff9700 1 -- 172.21.15.89:0/3781414678 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f4058400 con 0x563571330490 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f94ecff9700 1 -- 172.21.15.89:0/3781414678 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f405c550 con 0x563571330490 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3781414678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 msgr2=0x563571326f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x563571326f50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f94f4087b60 tx=0x7f94f4087a90 comp rx=0 tx=0).stop 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3781414678 shutdown_connections 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571374510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x563571326f50 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/3781414678 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x56357132e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.084+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3781414678 >> 172.21.15.89:0/3781414678 conn(0x563571261310 msgr2=0x56357137aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.085+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3781414678 shutdown_connections 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.085+0000 7f9510e5eb80 1 -- 172.21.15.89:0/3781414678 wait complete. 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.085+0000 7f9510e5eb80 1 -- start start 2023-11-24T06:20:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.085+0000 7f9510e5eb80 1 Processor -- start 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x5635713766c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571360970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f9510e5eb80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571360eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357135dfd0 con 0x563571330490 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635713e2d70 con 0x563571348ed0 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f9510e5eb80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x56357135e1e0 con 0x563571333080 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f9510e5eb80 10 client.0 _ll_register_callbacks cb 0x56357116ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fe59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x5635713766c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fe59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x5635713766c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38134/0 (socket says 172.21.15.89:38134) 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fe59c700 1 -- 172.21.15.89:0/1839108991 learned_addr learned my addr 172.21.15.89:0/1839108991 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571360970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fed9d700 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571360eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fe59c700 1 -- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 msgr2=0x563571360970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fe59c700 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571360970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fe59c700 1 -- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 msgr2=0x563571360eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fe59c700 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571360eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fe59c700 1 -- 172.21.15.89:0/1839108991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9500143060 con 0x563571330490 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.086+0000 7f94fed9d700 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571360eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94fe59c700 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x5635713766c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f94f4058100 tx=0x7f94f4089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6291235e430f0085 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94dd7fa700 1 -- 172.21.15.89:0/1839108991 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4086040 con 0x563571330490 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94dd7fa700 1 -- 172.21.15.89:0/1839108991 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94f4060700 con 0x563571330490 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 1 -- 172.21.15.89:0/1839108991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94f000ee60 con 0x563571330490 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94dd7fa700 1 -- 172.21.15.89:0/1839108991 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94f405c550 con 0x563571330490 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 10 client.5757 fetch_fsmap learned FSMap version 31 2023-11-24T06:20:33.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 1 -- 172.21.15.89:0/1839108991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x5635710d6200 con 0x563571330490 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94dd7fa700 1 -- 172.21.15.89:0/1839108991 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f000ee60 con 0x563571330490 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94dd7fa700 1 -- 172.21.15.89:0/1839108991 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 31) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f94f4076070 con 0x563571330490 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 10 client.5757 fetch_fsmap finished waiting for FSMap version 31 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 10 client.5757 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 20 client.5757 populate_metadata read hostname 'smithi089' 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] conn(0x563571348320 0x563571348700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 4 client.5757 mds_command: new command op to 5577 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f9510e5eb80 1 -- 172.21.15.89:0/1839108991 --> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56357124cf80 con 0x563571348320 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] conn(0x563571348320 0x563571348700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.087+0000 7f94dd7fa700 10 client.5757 ms_handle_connect on v2:172.21.15.89:6835/1341821445 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f94fdd9b700 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] conn(0x563571348320 0x563571348700 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcd4c0b9f421b2a6 server_cookie=85b9516a7c3d5cff in_seq=0 out_seq=0 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f94dd7fa700 1 -- 172.21.15.89:0/1839108991 <== mds.0 v2:172.21.15.89:6835/1341821445 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56357124cf80 con 0x563571348320 2023-11-24T06:20:33.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f94dd7fa700 10 client.5757 handle_command_reply: tid=0 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 client.5757 shutdown 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 -- 172.21.15.89:0/1839108991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 msgr2=0x5635713766c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x5635713766c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f94f4058100 tx=0x7f94f4089040 comp rx=0 tx=0).stop 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 -- 172.21.15.89:0/1839108991 shutdown_connections 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x563571348ed0 0x563571360eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.89:6835/1341821445,v1:172.21.15.89:6837/1341821445] conn(0x563571348320 0x563571348700 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563571330490 0x5635713766c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 --2- 172.21.15.89:0/1839108991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x563571333080 0x563571360970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f94dd7fa700 0 client.5757 ms_handle_reset on v2:172.21.15.89:6835/1341821445 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 -- 172.21.15.89:0/1839108991 >> 172.21.15.89:0/1839108991 conn(0x563571261310 msgr2=0x56357137aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 -- 172.21.15.89:0/1839108991 shutdown_connections 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 1 -- 172.21.15.89:0/1839108991 wait complete. 2023-11-24T06:20:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.088+0000 7f9510e5eb80 20 client.5757 trim_cache size 0 max 16384 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "be2488f4-68a7-42cc-93d1-ae9c6921fb48", 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 5577, 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 10.444787635999999, 2023-11-24T06:20:33.100 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 31, 2023-11-24T06:20:33.101 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 53, 2023-11-24T06:20:33.101 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 53, 2023-11-24T06:20:33.101 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 15.407490076 2023-11-24T06:20:33.101 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 -- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94e006aed0 msgr2=0x7f94e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94e006aed0 0x7f94e006d390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f94f00090b0 tx=0x7f94f001e040 comp rx=0 tx=0).stop 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 -- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 msgr2=0x7f95001526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f95001526a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f94f40072c0 tx=0x7f94f4007680 comp rx=0 tx=0).stop 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 -- 172.21.15.89:0/1595508764 shutdown_connections 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f94e006aed0 0x7f94e006d390 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950015dea0 0x7f9500136500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e8003fb0 0x7f95001526a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 --2- 172.21.15.89:0/1595508764 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9500142be0 0x7f950015d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 -- 172.21.15.89:0/1595508764 >> 172.21.15.89:0/1595508764 conn(0x7f9500071b90 msgr2=0x7f950015e710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 -- 172.21.15.89:0/1595508764 shutdown_connections 2023-11-24T06:20:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.089+0000 7f9504c65700 1 -- 172.21.15.89:0/1595508764 wait complete. 2023-11-24T06:20:33.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:20:33.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.221+0000 7fd6b1559700 1 Processor -- start 2023-11-24T06:20:33.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.221+0000 7fd6b1559700 1 -- start start 2023-11-24T06:20:33.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.221+0000 7fd6b1559700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd6ac13ea80 0x7fd6ac13ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.221+0000 7fd6b1559700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac160580 0x7fd6ac0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.221+0000 7fd6b1559700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6ac0ccce0 0x7fd6ac141590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.221+0000 7fd6b1559700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac0cd170 con 0x7fd6ac0ccce0 2023-11-24T06:20:33.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.221+0000 7fd6b1559700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac06e9f0 con 0x7fd6ac13ea80 2023-11-24T06:20:33.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.221+0000 7fd6b1559700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac142450 con 0x7fd6ac160580 2023-11-24T06:20:33.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6ac0ccce0 0x7fd6ac141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6ac0ccce0 0x7fd6ac141590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38146/0 (socket says 172.21.15.89:38146) 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 -- 172.21.15.89:0/2954719780 learned_addr learned my addr 172.21.15.89:0/2954719780 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6aaffd700 1 --2- 172.21.15.89:0/2954719780 >> v2:172.21.15.123:3300/0 conn(0x7fd6ac13ea80 0x7fd6ac13ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 -- 172.21.15.89:0/2954719780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac160580 msgr2=0x7fd6ac0cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 --2- 172.21.15.89:0/2954719780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac160580 0x7fd6ac0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 -- 172.21.15.89:0/2954719780 >> v2:172.21.15.123:3300/0 conn(0x7fd6ac13ea80 msgr2=0x7fd6ac13ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 --2- 172.21.15.89:0/2954719780 >> v2:172.21.15.123:3300/0 conn(0x7fd6ac13ea80 0x7fd6ac13ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 -- 172.21.15.89:0/2954719780 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac142170 con 0x7fd6ac0ccce0 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6ab7fe700 1 --2- 172.21.15.89:0/2954719780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0ccce0 0x7fd6ac141590 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac072ea0 tx=0x7fd6a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe999eba34a528b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6a9ffb700 1 -- 172.21.15.89:0/2954719780 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400e070 con 0x7fd6ac0ccce0 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6a9ffb700 1 -- 172.21.15.89:0/2954719780 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a400b9c0 con 0x7fd6ac0ccce0 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.222+0000 7fd6a9ffb700 1 -- 172.21.15.89:0/2954719780 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400bd70 con 0x7fd6ac0ccce0 2023-11-24T06:20:33.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 -- 172.21.15.89:0/2954719780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0ccce0 msgr2=0x7fd6ac141590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2954719780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0ccce0 0x7fd6ac141590 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac072ea0 tx=0x7fd6a400b330 comp rx=0 tx=0).stop 2023-11-24T06:20:33.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 -- 172.21.15.89:0/2954719780 shutdown_connections 2023-11-24T06:20:33.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2954719780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0ccce0 0x7fd6ac141590 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2954719780 >> v2:172.21.15.123:3300/0 conn(0x7fd6ac13ea80 0x7fd6ac13ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2954719780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac160580 0x7fd6ac0cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 -- 172.21.15.89:0/2954719780 >> 172.21.15.89:0/2954719780 conn(0x7fd6ac06b1d0 msgr2=0x7fd6ac06b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:33.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 -- 172.21.15.89:0/2954719780 shutdown_connections 2023-11-24T06:20:33.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.223+0000 7fd6b1559700 1 -- 172.21.15.89:0/2954719780 wait complete. 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.224+0000 7fd6b1559700 1 Processor -- start 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.224+0000 7fd6b1559700 1 -- start start 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6b1559700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6ac0ccce0 0x7fd6ac13e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6b1559700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac13ea80 0x7fd6ac159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6b1559700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac160580 0x7fd6ac0cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6b1559700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac13e910 con 0x7fd6ac160580 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6b1559700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac154580 con 0x7fd6ac0ccce0 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6b1559700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac1553e0 con 0x7fd6ac13ea80 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac160580 0x7fd6ac0cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac160580 0x7fd6ac0cea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38152/0 (socket says 172.21.15.89:38152) 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 -- 172.21.15.89:0/2878737061 learned_addr learned my addr 172.21.15.89:0/2878737061 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6aaffd700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6ac0ccce0 0x7fd6ac13e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6aa7fc700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac13ea80 0x7fd6ac159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 -- 172.21.15.89:0/2878737061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac13ea80 msgr2=0x7fd6ac159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac13ea80 0x7fd6ac159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 -- 172.21.15.89:0/2878737061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6ac0ccce0 msgr2=0x7fd6ac13e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6ac0ccce0 0x7fd6ac13e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 -- 172.21.15.89:0/2878737061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a4009b30 con 0x7fd6ac160580 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.225+0000 7fd6ab7fe700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac160580 0x7fd6ac0cea90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac072f70 tx=0x7fd6a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.226+0000 7fd6937fe700 1 -- 172.21.15.89:0/2878737061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40099b0 con 0x7fd6ac160580 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.226+0000 7fd6937fe700 1 -- 172.21.15.89:0/2878737061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a4004250 con 0x7fd6ac160580 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.226+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac1547d0 con 0x7fd6ac160580 2023-11-24T06:20:33.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.226+0000 7fd6937fe700 1 -- 172.21.15.89:0/2878737061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4004600 con 0x7fd6ac160580 2023-11-24T06:20:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.226+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac154d90 con 0x7fd6ac160580 2023-11-24T06:20:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.227+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac0cf8b0 con 0x7fd6ac160580 2023-11-24T06:20:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.227+0000 7fd6937fe700 1 -- 172.21.15.89:0/2878737061 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd6a4004760 con 0x7fd6ac160580 2023-11-24T06:20:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.227+0000 7fd6937fe700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd69806aed0 0x7fd69806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.227+0000 7fd6937fe700 1 -- 172.21.15.89:0/2878737061 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6a408aae0 con 0x7fd6ac160580 2023-11-24T06:20:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.227+0000 7fd6aaffd700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd69806aed0 0x7fd69806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:33.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.228+0000 7fd6aaffd700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd69806aed0 0x7fd69806d390 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd69c006b20 tx=0x7fd69c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:33.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.232+0000 7fd6937fe700 1 -- 172.21.15.89:0/2878737061 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6a4055b80 con 0x7fd6ac160580 2023-11-24T06:20:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.466+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd6ac157b10 con 0x7fd6ac160580 2023-11-24T06:20:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.466+0000 7fd6937fe700 1 -- 172.21.15.89:0/2878737061 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v31) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd6a4020020 con 0x7fd6ac160580 2023-11-24T06:20:33.469 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:20:33.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd69806aed0 msgr2=0x7fd69806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd69806aed0 0x7fd69806d390 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd69c006b20 tx=0x7fd69c008040 comp rx=0 tx=0).stop 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac160580 msgr2=0x7fd6ac0cea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac160580 0x7fd6ac0cea90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac072f70 tx=0x7fd6a4004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 shutdown_connections 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac160580 0x7fd6ac0cea90 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd69806aed0 0x7fd69806d390 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6ac0ccce0 0x7fd6ac13e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 --2- 172.21.15.89:0/2878737061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6ac13ea80 0x7fd6ac159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 >> 172.21.15.89:0/2878737061 conn(0x7fd6ac06b1d0 msgr2=0x7fd6ac06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 shutdown_connections 2023-11-24T06:20:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:33.468+0000 7fd6b1559700 1 -- 172.21.15.89:0/2878737061 wait complete. 2023-11-24T06:20:33.483 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T06:20:33.483 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:20:33.484 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2023-11-24T06:20:33.484 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T06:20:33.484 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T06:20:33.484 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T06:20:33.484 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T06:20:33.484 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:20:33.484 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:20:33.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:33.484 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2023-11-24T06:20:33.502 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:20:33.502 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:20:33.502 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:20:33.502 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:33.503 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-11-24T06:20:33.503 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:33.503 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:20:33.503 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:33.503 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42326sec preferred_lft 42326sec 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4376:7845:d8a4:7bc1/64 scope link noprefixroute 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:33.504 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 12:de:17:7b:11:37 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::3cfa:7aff:fed2:90c8/64 scope link 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 12:de:17:7b:11:37 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::10de:17ff:fe7b:1137/64 scope link 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 76:9a:20:1a:a1:88 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::749a:20ff:fe1a:a188/64 scope link 2023-11-24T06:20:33.505 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:33.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:33.506 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:20:33.567 INFO:teuthology.orchestra.run.smithi089.stdout:1 2023-11-24T06:20:33.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:33.568 DEBUG:teuthology.orchestra.run.smithi089:> route 2023-11-24T06:20:33.629 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2023-11-24T06:20:33.629 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:20:33.629 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:20:33.629 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:20:33.629 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:20:33.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:33.629 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:20:33.630 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:20:33.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:20:33.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:20:33.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:20:33.630 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:20:33.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:33.733 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:20:33.750 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T06:20:33.750 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T06:20:33.750 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:20:33.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:33.750 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2023-11-24T06:20:33.816 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:20:33.816 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T06:20:33.873 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:20:33.874 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:20:33.874 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:20:33.930 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:20:33.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:33.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:20:34.016 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:20:34.016 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:20:34.017 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:20:34.025 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:20:34.025 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:20:34.025 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:20:34.025 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:20:34.025 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:20:34.051 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:20:34.051 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:20:34.051 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:20:34.051 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:20:34.052 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:20:34.052 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:20:34.052 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:20:34.052 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:20:34.068 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:20:34.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:34.069 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:20:34.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:34.095 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:20:34.177 INFO:teuthology.orchestra.run.smithi089.stdout:client5341 2023-11-24T06:20:34.178 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:34.178 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:34.236 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:20:34.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.393+0000 7f9b467d8700 1 Processor -- start 2023-11-24T06:20:34.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.393+0000 7f9b467d8700 1 -- start start 2023-11-24T06:20:34.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b467d8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 0x7f9b400712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:34.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b467d8700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b400b3000 con 0x7f9b40158770 2023-11-24T06:20:34.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b467d8700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b400b32e0 con 0x7f9b4006eb70 2023-11-24T06:20:34.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b467d8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b400b35c0 con 0x7f9b4015b140 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b3ffff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9b40158770 0x7f9b4006e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41664/0 (socket says 172.21.15.89:41664) 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b3ffff700 1 -- 172.21.15.89:0/557745935 learned_addr learned my addr 172.21.15.89:0/557745935 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b44d75700 1 --2- 172.21.15.89:0/557745935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 0x7f9b400712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b3effd700 1 -- 172.21.15.89:0/557745935 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547169367 0 0) 0x7f9b400b3000 con 0x7f9b40158770 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b3effd700 1 -- 172.21.15.89:0/557745935 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b280035b0 con 0x7f9b40158770 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b3effd700 1 -- 172.21.15.89:0/557745935 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795989198 0 0) 0x7f9b400b32e0 con 0x7f9b4006eb70 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b44d75700 1 -- 172.21.15.89:0/557745935 >> v1:172.21.15.123:6789/0 conn(0x7f9b4006eb70 legacy=0x7f9b40070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b44d75700 1 -- 172.21.15.89:0/557745935 >> v1:172.21.15.89:6789/0 conn(0x7f9b40158770 legacy=0x7f9b4006e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.394+0000 7f9b44d75700 1 -- 172.21.15.89:0/557745935 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b400b38a0 con 0x7f9b4015b140 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b44d75700 1 --2- 172.21.15.89:0/557745935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 0x7f9b400712d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009dc0 tx=0x7f9b3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6001bff1a5bbbb08 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b3effd700 1 -- 172.21.15.89:0/557745935 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800e070 con 0x7f9b4015b140 2023-11-24T06:20:34.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b3effd700 1 -- 172.21.15.89:0/557745935 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b38003a20 con 0x7f9b4015b140 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b3effd700 1 -- 172.21.15.89:0/557745935 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34001670 con 0x7f9b4015b140 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b467d8700 1 -- 172.21.15.89:0/557745935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 msgr2=0x7f9b400712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b467d8700 1 --2- 172.21.15.89:0/557745935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 0x7f9b400712d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009dc0 tx=0x7f9b3800b330 comp rx=0 tx=0).stop 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b467d8700 1 -- 172.21.15.89:0/557745935 shutdown_connections 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b467d8700 1 --2- 172.21.15.89:0/557745935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 0x7f9b400712d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b467d8700 1 -- 172.21.15.89:0/557745935 >> 172.21.15.89:0/557745935 conn(0x7f9b400b1740 msgr2=0x7f9b400b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b467d8700 1 -- 172.21.15.89:0/557745935 shutdown_connections 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.395+0000 7f9b467d8700 1 -- 172.21.15.89:0/557745935 wait complete. 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.397+0000 7f9b467d8700 1 Processor -- start 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.397+0000 7f9b467d8700 1 -- start start 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b467d8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4006eb70 0x7f9b40145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:34.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b467d8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b40158770 0x7f9b40149030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b467d8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 0x7f9b401f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b467d8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40148650 con 0x7f9b4006eb70 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b467d8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b401f5ea0 con 0x7f9b40158770 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b467d8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40148ba0 con 0x7f9b4015b140 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4006eb70 0x7f9b40145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4006eb70 0x7f9b40145e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38166/0 (socket says 172.21.15.89:38166) 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 -- 172.21.15.89:0/2974293047 learned_addr learned my addr 172.21.15.89:0/2974293047 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 -- 172.21.15.89:0/2974293047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 msgr2=0x7f9b401f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 0x7f9b401f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 -- 172.21.15.89:0/2974293047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b40158770 msgr2=0x7f9b40149030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b40158770 0x7f9b40149030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:34.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 -- 172.21.15.89:0/2974293047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b38009a70 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.398+0000 7f9b3ffff700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4006eb70 0x7f9b40145e00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c750 tx=0x7f9b3400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.399+0000 7f9b3cff9700 1 -- 172.21.15.89:0/2974293047 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34004c80 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.399+0000 7f9b3cff9700 1 -- 172.21.15.89:0/2974293047 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b34004de0 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.399+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b40146340 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.399+0000 7f9b3cff9700 1 -- 172.21.15.89:0/2974293047 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34007310 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.399+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b401f5bc0 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.400+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b40074e80 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.400+0000 7f9b3cff9700 1 -- 172.21.15.89:0/2974293047 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9b34007470 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.400+0000 7f9b3cff9700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b2c06aed0 0x7f9b2c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.400+0000 7f9b3cff9700 1 -- 172.21.15.89:0/2974293047 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b34088a80 con 0x7f9b4006eb70 2023-11-24T06:20:34.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.401+0000 7f9b3f7fe700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b2c06aed0 0x7f9b2c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:34.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.401+0000 7f9b3f7fe700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b2c06aed0 0x7f9b2c06d390 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9b400a9630 tx=0x7f9b300079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:34.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.405+0000 7f9b3cff9700 1 -- 172.21.15.89:0/2974293047 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b40074e80 con 0x7f9b4006eb70 2023-11-24T06:20:34.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.641+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9b40074e80 con 0x7f9b4006eb70 2023-11-24T06:20:34.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.642+0000 7f9b3cff9700 1 -- 172.21.15.89:0/2974293047 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v31) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9b40074e80 con 0x7f9b4006eb70 2023-11-24T06:20:34.645 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b2c06aed0 msgr2=0x7f9b2c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b2c06aed0 0x7f9b2c06d390 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9b400a9630 tx=0x7f9b300079c0 comp rx=0 tx=0).stop 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4006eb70 msgr2=0x7f9b40145e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4006eb70 0x7f9b40145e00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c750 tx=0x7f9b3400cb10 comp rx=0 tx=0).stop 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 shutdown_connections 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b4015b140 0x7f9b401f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4006eb70 0x7f9b40145e00 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b2c06aed0 0x7f9b2c06d390 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:34.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 --2- 172.21.15.89:0/2974293047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b40158770 0x7f9b40149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:34.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 >> 172.21.15.89:0/2974293047 conn(0x7f9b400b1740 msgr2=0x7f9b4015c200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:34.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 shutdown_connections 2023-11-24T06:20:34.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:34.644+0000 7f9b467d8700 1 -- 172.21.15.89:0/2974293047 wait complete. 2023-11-24T06:20:34.659 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T06:20:34.659 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:20:34.659 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2023-11-24T06:20:34.660 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T06:20:34.660 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T06:20:34.660 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T06:20:34.660 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T06:20:34.660 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:20:34.660 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:20:34.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:34.661 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2023-11-24T06:20:34.683 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:20:34.683 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:20:34.683 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:20:34.683 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:34.683 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42317sec preferred_lft 42317sec 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::a981:dd7b:bd0d:851a/64 scope link noprefixroute 2023-11-24T06:20:34.684 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 1a:69:ea:ee:dc:d5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::682d:1bff:fea0:f45c/64 scope link 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether d2:06:87:3a:6b:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::d006:87ff:fe3a:6bfb/64 scope link 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:20:34.685 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 1a:69:ea:ee:dc:d5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T06:20:34.686 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::1869:eaff:feee:dcd5/64 scope link 2023-11-24T06:20:34.686 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:20:34.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:34.686 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:20:34.755 INFO:teuthology.orchestra.run.smithi123.stdout:1 2023-11-24T06:20:34.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:34.757 DEBUG:teuthology.orchestra.run.smithi123:> route 2023-11-24T06:20:34.821 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2023-11-24T06:20:34.821 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:20:34.821 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:20:34.821 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:20:34.821 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:20:34.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:34.822 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:20:34.822 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:20:34.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:20:34.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:20:34.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:20:34.822 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:20:34.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:34.944 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:20:34.967 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T06:20:34.967 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T06:20:34.967 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:20:34.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:34.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2023-11-24T06:20:35.042 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:20:35.042 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T06:20:35.103 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:20:35.103 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:20:35.103 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:20:35.163 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:20:35.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:35.164 DEBUG:teuthology.orchestra.run.smithi123:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:20:35.259 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:20:35.259 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:20:35.259 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:20:35.269 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:20:35.269 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:20:35.269 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:20:35.269 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:20:35.270 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:20:35.312 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:20:35.313 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:20:35.313 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:20:35.313 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:20:35.313 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:20:35.313 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:20:35.313 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:20:35.313 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:20:35.315 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:20:35.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:35.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:20:35.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:35.344 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:20:35.416 INFO:teuthology.orchestra.run.smithi123.stdout:client5347 2023-11-24T06:20:35.417 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:35.418 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:35.474 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T06:20:35.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.608+0000 7f7362dce700 1 Processor -- start 2023-11-24T06:20:35.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.608+0000 7f7362dce700 1 -- start start 2023-11-24T06:20:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f7362dce700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f735c145620 0x7f735c06dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f7362dce700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c0cd810 0x7f735c06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f7362dce700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c0cf530 con 0x7f735c073f90 2023-11-24T06:20:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f7362dce700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735c06b8f0 con 0x7f735c145620 2023-11-24T06:20:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f7362dce700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c06bbd0 con 0x7f735c0cd810 2023-11-24T06:20:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f736136b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f735c073f90 0x7f735c0cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41682/0 (socket says 172.21.15.89:41682) 2023-11-24T06:20:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f736136b700 1 -- 172.21.15.89:0/1090181757 learned_addr learned my addr 172.21.15.89:0/1090181757 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f7360b6a700 1 --2- 172.21.15.89:0/1090181757 >> v2:172.21.15.123:3300/0 conn(0x7f735c145620 0x7f735c06dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f735bfff700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c0cd810 0x7f735c06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872894498 0 0) 0x7f735c0cf530 con 0x7f735c073f90 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.609+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734c0035b0 con 0x7f735c073f90 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965952113 0 0) 0x7f734c0035b0 con 0x7f735c073f90 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c0cd810 msgr2=0x7f735c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c0cd810 0x7f735c06aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 >> v2:172.21.15.123:3300/0 conn(0x7f735c145620 msgr2=0x7f735c06dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 --2- 172.21.15.89:0/1090181757 >> v2:172.21.15.123:3300/0 conn(0x7f735c145620 0x7f735c06dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0cf1c0 con 0x7f735c073f90 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f73540019f0 con 0x7f735c073f90 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f7360b6a700 1 --2- 172.21.15.89:0/1090181757 >> v2:172.21.15.123:3300/0 conn(0x7f735c145620 0x7f735c06dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 >> v1:172.21.15.89:6789/0 conn(0x7f735c073f90 legacy=0x7f735c0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f734c003a00 0x7f734c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c0cd810 0x7f734c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f734c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c06b8f0 con 0x7f734c004d90 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c06bbd0 con 0x7f734c003a00 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c00ba10 con 0x7f735c0cd810 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f736136b700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f734c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f7360b6a700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f734c003a00 0x7f734c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735bfff700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c0cd810 0x7f734c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f736136b700 1 -- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c0cd810 msgr2=0x7f734c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f736136b700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c0cd810 0x7f734c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f736136b700 1 -- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f734c003a00 msgr2=0x7f734c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f736136b700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f734c003a00 0x7f734c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f736136b700 1 -- 172.21.15.89:0/1090181757 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c0035b0 con 0x7f734c004d90 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f736136b700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f734c005210 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f735400a860 tx=0x7f734c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2f81db292f4cfa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354012600 con 0x7f734c004d90 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7354012780 con 0x7f734c004d90 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.610+0000 7f735b7fe700 1 -- 172.21.15.89:0/1090181757 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354012b30 con 0x7f734c004d90 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.611+0000 7f7362dce700 1 -- 172.21.15.89:0/1090181757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 msgr2=0x7f734c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.611+0000 7f7362dce700 1 --2- 172.21.15.89:0/1090181757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f734c005210 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f735400a860 tx=0x7f734c005880 comp rx=0 tx=0).stop 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.611+0000 7f736136b700 1 -- 172.21.15.89:0/1090181757 reap_dead start 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.611+0000 7f7362dce700 1 -- 172.21.15.89:0/1090181757 shutdown_connections 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.611+0000 7f7362dce700 1 -- 172.21.15.89:0/1090181757 >> 172.21.15.89:0/1090181757 conn(0x7f735c0d0b40 msgr2=0x7f735c0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.611+0000 7f7362dce700 1 -- 172.21.15.89:0/1090181757 shutdown_connections 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.611+0000 7f7362dce700 1 -- 172.21.15.89:0/1090181757 wait complete. 2023-11-24T06:20:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.612+0000 7f7362dce700 1 Processor -- start 2023-11-24T06:20:35.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.612+0000 7f7362dce700 1 -- start start 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7362dce700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f735c1f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7362dce700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f735c1f51b0 0x7f735c071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7362dce700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c072100 0x7f735c1f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7362dce700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c072590 con 0x7f734c004d90 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7362dce700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c143ea0 con 0x7f735c1f51b0 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7362dce700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1f57e0 con 0x7f735c072100 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7360b6a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f735c1f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7360b6a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f735c1f4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38186/0 (socket says 172.21.15.89:38186) 2023-11-24T06:20:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f7360b6a700 1 -- 172.21.15.89:0/4232917500 learned_addr learned my addr 172.21.15.89:0/4232917500 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f735bfff700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f735c1f51b0 0x7f735c071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.613+0000 7f736136b700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c072100 0x7f735c1f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f7360b6a700 1 -- 172.21.15.89:0/4232917500 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c072100 msgr2=0x7f735c1f0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f7360b6a700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c072100 0x7f735c1f0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f7360b6a700 1 -- 172.21.15.89:0/4232917500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f735c1f51b0 msgr2=0x7f735c071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f7360b6a700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f735c1f51b0 0x7f735c071bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f7360b6a700 1 -- 172.21.15.89:0/4232917500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c1f5ac0 con 0x7f734c004d90 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f7360b6a700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f735c1f4c70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7350011cd0 tx=0x7f7350009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f73597fa700 1 -- 172.21.15.89:0/4232917500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735000f9a0 con 0x7f734c004d90 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f73597fa700 1 -- 172.21.15.89:0/4232917500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735000fb00 con 0x7f734c004d90 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c0c5ac0 con 0x7f734c004d90 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f73597fa700 1 -- 172.21.15.89:0/4232917500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735000feb0 con 0x7f734c004d90 2023-11-24T06:20:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.614+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c0c6080 con 0x7f734c004d90 2023-11-24T06:20:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.615+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735c067f60 con 0x7f734c004d90 2023-11-24T06:20:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.616+0000 7f73597fa700 1 -- 172.21.15.89:0/4232917500 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7350004030 con 0x7f734c004d90 2023-11-24T06:20:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.617+0000 7f73597fa700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f734006aed0 0x7f734006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.617+0000 7f73597fa700 1 -- 172.21.15.89:0/4232917500 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7350089f10 con 0x7f734c004d90 2023-11-24T06:20:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.617+0000 7f735bfff700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f734006aed0 0x7f734006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.617+0000 7f735bfff700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f734006aed0 0x7f734006d390 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7348009c60 tx=0x7f7348009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.621+0000 7f73597fa700 1 -- 172.21.15.89:0/4232917500 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7350055030 con 0x7f734c004d90 2023-11-24T06:20:35.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:35.853+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f735c0b1870 con 0x7f734c004d90 2023-11-24T06:20:36.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.008+0000 7f73597fa700 1 -- 172.21.15.89:0/4232917500 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v32) v1 ==== 136+0+0 (secure 0 0 0) 0x7f73500220c0 con 0x7f734c004d90 2023-11-24T06:20:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f734006aed0 msgr2=0x7f734006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f734006aed0 0x7f734006d390 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7348009c60 tx=0x7f7348009210 comp rx=0 tx=0).stop 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 msgr2=0x7f735c1f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f735c1f4c70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7350011cd0 tx=0x7f7350009ed0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 shutdown_connections 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f735c072100 0x7f735c1f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c004d90 0x7f735c1f4c70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f734006aed0 0x7f734006d390 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 --2- 172.21.15.89:0/4232917500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f735c1f51b0 0x7f735c071bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 >> 172.21.15.89:0/4232917500 conn(0x7f735c0d0b40 msgr2=0x7f735c06f480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 shutdown_connections 2023-11-24T06:20:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.010+0000 7f7362dce700 1 -- 172.21.15.89:0/4232917500 wait complete. 2023-11-24T06:20:36.025 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T06:20:36.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.145+0000 7fce24e15700 1 Processor -- start 2023-11-24T06:20:36.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.145+0000 7fce24e15700 1 -- start start 2023-11-24T06:20:36.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce24e15700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 0x7fce200b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce24e15700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200d2930 con 0x7fce200d45d0 2023-11-24T06:20:36.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce24e15700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200d2c10 con 0x7fce2015e140 2023-11-24T06:20:36.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce24e15700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2015e520 con 0x7fce200d39d0 2023-11-24T06:20:36.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1dd9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fce200d45d0 0x7fce200b70d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41698/0 (socket says 172.21.15.89:41698) 2023-11-24T06:20:36.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1dd9b700 1 -- 172.21.15.89:0/618238007 learned_addr learned my addr 172.21.15.89:0/618238007 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:36.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1ed9d700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 0x7fce200b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719643974 0 0) 0x7fce200d2930 con 0x7fce200d45d0 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce080035b0 con 0x7fce200d45d0 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056808847 0 0) 0x7fce200d2c10 con 0x7fce2015e140 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce200d2930 con 0x7fce2015e140 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4284575240 0 0) 0x7fce080035b0 con 0x7fce200d45d0 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 msgr2=0x7fce200b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 0x7fce200b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 >> v1:172.21.15.123:6789/0 conn(0x7fce2015e140 legacy=0x7fce200d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce200d2ef0 con 0x7fce200d45d0 2023-11-24T06:20:36.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1ed9d700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 0x7fce200b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.146+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fce14001670 con 0x7fce200d45d0 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 >> v1:172.21.15.89:6789/0 conn(0x7fce200d45d0 legacy=0x7fce200b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 0x7fce08003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08004100 0x7fce08004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce08004d90 0x7fce08005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2015e520 con 0x7fce08004100 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0800bac0 con 0x7fce08004d90 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0800bc80 con 0x7fce200d39d0 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1dd9b700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08004100 0x7fce08004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1e59c700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 0x7fce08003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1ed9d700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce08004d90 0x7fce08005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1dd9b700 1 -- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 msgr2=0x7fce08003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1dd9b700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 0x7fce08003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1dd9b700 1 -- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce08004d90 msgr2=0x7fce08005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1dd9b700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce08004d90 0x7fce08005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1dd9b700 1 -- 172.21.15.89:0/618238007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce18009a70 con 0x7fce08004100 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1e59c700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce200d39d0 0x7fce08003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1dd9b700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08004100 0x7fce08004850 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fce1400c210 tx=0x7fce08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0416990ec28a1f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1400cdb0 con 0x7fce08004100 2023-11-24T06:20:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce14001af0 con 0x7fce08004100 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.147+0000 7fce1d59a700 1 -- 172.21.15.89:0/618238007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce14005ba0 con 0x7fce08004100 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.148+0000 7fce24e15700 1 -- 172.21.15.89:0/618238007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08004100 msgr2=0x7fce08004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.148+0000 7fce24e15700 1 --2- 172.21.15.89:0/618238007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08004100 0x7fce08004850 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fce1400c210 tx=0x7fce08005880 comp rx=0 tx=0).stop 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.148+0000 7fce1ed9d700 1 -- 172.21.15.89:0/618238007 reap_dead start 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.148+0000 7fce24e15700 1 -- 172.21.15.89:0/618238007 shutdown_connections 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.148+0000 7fce24e15700 1 -- 172.21.15.89:0/618238007 >> 172.21.15.89:0/618238007 conn(0x7fce2006b410 msgr2=0x7fce2006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.148+0000 7fce24e15700 1 -- 172.21.15.89:0/618238007 shutdown_connections 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.148+0000 7fce24e15700 1 -- 172.21.15.89:0/618238007 wait complete. 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.149+0000 7fce24e15700 1 Processor -- start 2023-11-24T06:20:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.149+0000 7fce24e15700 1 -- start start 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce24e15700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce08004100 0x7fce201537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce24e15700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce201f76c0 0x7fce200b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce24e15700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce200b4e10 0x7fce200c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce24e15700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce20153d20 con 0x7fce201f76c0 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce24e15700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce200c9560 con 0x7fce200b4e10 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce24e15700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce200c8950 con 0x7fce08004100 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce201f76c0 0x7fce200b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce201f76c0 0x7fce200b48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38206/0 (socket says 172.21.15.89:38206) 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 -- 172.21.15.89:0/3542448274 learned_addr learned my addr 172.21.15.89:0/3542448274 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1e59c700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce08004100 0x7fce201537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1ed9d700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce200b4e10 0x7fce200c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 -- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce08004100 msgr2=0x7fce201537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce08004100 0x7fce201537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 -- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce200b4e10 msgr2=0x7fce200c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce200b4e10 0x7fce200c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 -- 172.21.15.89:0/3542448274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce200c8c30 con 0x7fce201f76c0 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1ed9d700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce200b4e10 0x7fce200c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.150+0000 7fce1dd9b700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce201f76c0 0x7fce200b48d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fce1400c800 tx=0x7fce14001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.151+0000 7fce0effd700 1 -- 172.21.15.89:0/3542448274 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce140072d0 con 0x7fce201f76c0 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.151+0000 7fce0effd700 1 -- 172.21.15.89:0/3542448274 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce14007450 con 0x7fce201f76c0 2023-11-24T06:20:36.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.151+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce200c8f10 con 0x7fce201f76c0 2023-11-24T06:20:36.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.151+0000 7fce0effd700 1 -- 172.21.15.89:0/3542448274 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce14007800 con 0x7fce201f76c0 2023-11-24T06:20:36.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.151+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce201f70d0 con 0x7fce201f76c0 2023-11-24T06:20:36.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.152+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce200d1b90 con 0x7fce201f76c0 2023-11-24T06:20:36.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.152+0000 7fce0effd700 1 -- 172.21.15.89:0/3542448274 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fce1401e430 con 0x7fce201f76c0 2023-11-24T06:20:36.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.152+0000 7fce0effd700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce0006aed0 0x7fce0006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.153+0000 7fce0effd700 1 -- 172.21.15.89:0/3542448274 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce1408a1d0 con 0x7fce201f76c0 2023-11-24T06:20:36.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.153+0000 7fce1e59c700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce0006aed0 0x7fce0006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.153+0000 7fce1e59c700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce0006aed0 0x7fce0006d390 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fce10000c00 tx=0x7fce10009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.157+0000 7fce0effd700 1 -- 172.21.15.89:0/3542448274 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce14055310 con 0x7fce201f76c0 2023-11-24T06:20:36.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.376+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fce200d34b0 con 0x7fce0006aed0 2023-11-24T06:20:36.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.377+0000 7fce0effd700 1 -- 172.21.15.89:0/3542448274 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fce10001cd0 con 0x7fce0006aed0 2023-11-24T06:20:36.380 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-11-24T06:20:36.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:36.380 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2023-11-24T06:20:36.380 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:36.380 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-11-24T06:20:36.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce0006aed0 msgr2=0x7fce0006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce0006aed0 0x7fce0006d390 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fce10000c00 tx=0x7fce10009040 comp rx=0 tx=0).stop 2023-11-24T06:20:36.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce201f76c0 msgr2=0x7fce200b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce201f76c0 0x7fce200b48d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fce1400c800 tx=0x7fce14001980 comp rx=0 tx=0).stop 2023-11-24T06:20:36.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 shutdown_connections 2023-11-24T06:20:36.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fce0006aed0 0x7fce0006d390 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce200b4e10 0x7fce200c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce08004100 0x7fce201537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 --2- 172.21.15.89:0/3542448274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce201f76c0 0x7fce200b48d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 >> 172.21.15.89:0/3542448274 conn(0x7fce2006b410 msgr2=0x7fce200ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:36.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 shutdown_connections 2023-11-24T06:20:36.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.379+0000 7fce24e15700 1 -- 172.21.15.89:0/3542448274 wait complete. 2023-11-24T06:20:36.394 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T06:20:36.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.514+0000 7f8ba35fa700 1 Processor -- start 2023-11-24T06:20:36.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.515+0000 7f8ba35fa700 1 -- start start 2023-11-24T06:20:36.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.515+0000 7f8ba35fa700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8b9c15bf50 0x7f8b9c0ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.515+0000 7f8ba35fa700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0ce830 0x7f8b9c0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.515+0000 7f8ba35fa700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9c0d0140 con 0x7f8b9c0cf1f0 2023-11-24T06:20:36.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.515+0000 7f8ba35fa700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9c15c330 con 0x7f8b9c15bf50 2023-11-24T06:20:36.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.515+0000 7f8ba35fa700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c0d03e0 con 0x7f8b9c0ce830 2023-11-24T06:20:36.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.515+0000 7f8ba1b97700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8b9c0cf1f0 0x7f8b9c0c5c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41720/0 (socket says 172.21.15.89:41720) 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.515+0000 7f8ba1b97700 1 -- 172.21.15.89:0/4292351931 learned_addr learned my addr 172.21.15.89:0/4292351931 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba1396700 1 --2- 172.21.15.89:0/4292351931 >> v2:172.21.15.123:3300/0 conn(0x7f8b9c15bf50 0x7f8b9c0ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba0b95700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0ce830 0x7f8b9c0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088498686 0 0) 0x7f8b9c0d0140 con 0x7f8b9c0cf1f0 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b840035b0 con 0x7f8b9c0cf1f0 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4015495468 0 0) 0x7f8b840035b0 con 0x7f8b9c0cf1f0 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0ce830 msgr2=0x7f8b9c0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0ce830 0x7f8b9c0cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 >> v2:172.21.15.123:3300/0 conn(0x7f8b9c15bf50 msgr2=0x7f8b9c0ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 --2- 172.21.15.89:0/4292351931 >> v2:172.21.15.123:3300/0 conn(0x7f8b9c15bf50 0x7f8b9c0ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c0d06c0 con 0x7f8b9c0cf1f0 2023-11-24T06:20:36.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f8b8c0019f0 con 0x7f8b9c0cf1f0 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 >> v1:172.21.15.89:6789/0 conn(0x7f8b9c0cf1f0 legacy=0x7f8b9c0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b84003a00 0x7f8b84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0ce830 0x7f8b84004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b84005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c15c330 con 0x7f8b84004d90 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c0d03e0 con 0x7f8b84003a00 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8400ba10 con 0x7f8b9c0ce830 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba1b97700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b84005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba0b95700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0ce830 0x7f8b84004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba1396700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b84003a00 0x7f8b84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba1b97700 1 -- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0ce830 msgr2=0x7f8b84004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba1b97700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0ce830 0x7f8b84004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba1b97700 1 -- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b84003a00 msgr2=0x7f8b84003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba1b97700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b84003a00 0x7f8b84003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.516+0000 7f8ba1b97700 1 -- 172.21.15.89:0/4292351931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b840035b0 con 0x7f8b84004d90 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8ba1b97700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b84005210 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00a8d0 tx=0x7f8b84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366e891bcc1db13a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c00a340 con 0x7f8b84004d90 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c012600 con 0x7f8b84004d90 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8b9bfff700 1 -- 172.21.15.89:0/4292351931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0129b0 con 0x7f8b84004d90 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8ba35fa700 1 -- 172.21.15.89:0/4292351931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 msgr2=0x7f8b84005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8ba35fa700 1 --2- 172.21.15.89:0/4292351931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b84005210 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00a8d0 tx=0x7f8b84005880 comp rx=0 tx=0).stop 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8ba1b97700 1 -- 172.21.15.89:0/4292351931 reap_dead start 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8ba35fa700 1 -- 172.21.15.89:0/4292351931 shutdown_connections 2023-11-24T06:20:36.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8ba35fa700 1 -- 172.21.15.89:0/4292351931 >> 172.21.15.89:0/4292351931 conn(0x7f8b9c0b3780 msgr2=0x7f8b9c0b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:36.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8ba35fa700 1 -- 172.21.15.89:0/4292351931 shutdown_connections 2023-11-24T06:20:36.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.517+0000 7f8ba35fa700 1 -- 172.21.15.89:0/4292351931 wait complete. 2023-11-24T06:20:36.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba35fa700 1 Processor -- start 2023-11-24T06:20:36.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba35fa700 1 -- start start 2023-11-24T06:20:36.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba35fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b9c0caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba35fa700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0b4870 0x7f8b9c15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba35fa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b9c150f60 0x7f8b9c151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba35fa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c151950 con 0x7f8b84004d90 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba35fa700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c0b4d50 con 0x7f8b9c150f60 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba35fa700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c0b5030 con 0x7f8b9c0b4870 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba1396700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b9c0caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba1396700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b9c0caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38224/0 (socket says 172.21.15.89:38224) 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba1396700 1 -- 172.21.15.89:0/1568659864 learned_addr learned my addr 172.21.15.89:0/1568659864 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba1b97700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b9c150f60 0x7f8b9c151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.519+0000 7f8ba0b95700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0b4870 0x7f8b9c15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8ba1396700 1 -- 172.21.15.89:0/1568659864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0b4870 msgr2=0x7f8b9c15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8ba1396700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0b4870 0x7f8b9c15ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8ba1396700 1 -- 172.21.15.89:0/1568659864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b9c150f60 msgr2=0x7f8b9c151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8ba1396700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b9c150f60 0x7f8b9c151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8ba1396700 1 -- 172.21.15.89:0/1568659864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c0b5310 con 0x7f8b84004d90 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8ba1396700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b9c0caad0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8b94011aa0 tx=0x7f8b94011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8b99ffb700 1 -- 172.21.15.89:0/1568659864 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94011ed0 con 0x7f8b84004d90 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8b99ffb700 1 -- 172.21.15.89:0/1568659864 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9400f940 con 0x7f8b84004d90 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9c0b55f0 con 0x7f8b84004d90 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8b99ffb700 1 -- 172.21.15.89:0/1568659864 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9400fcd0 con 0x7f8b84004d90 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.520+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9c0cb010 con 0x7f8b84004d90 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.521+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9c068fb0 con 0x7f8b84004d90 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.522+0000 7f8b99ffb700 1 -- 172.21.15.89:0/1568659864 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8b94014440 con 0x7f8b84004d90 2023-11-24T06:20:36.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.522+0000 7f8b99ffb700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8b8806aed0 0x7f8b8806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.522+0000 7f8b99ffb700 1 -- 172.21.15.89:0/1568659864 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b94089c30 con 0x7f8b84004d90 2023-11-24T06:20:36.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.522+0000 7f8ba0b95700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8b8806aed0 0x7f8b8806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.522+0000 7f8ba0b95700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8b8806aed0 0x7f8b8806d390 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c1f4c40 tx=0x7f8b90009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.526+0000 7f8b99ffb700 1 -- 172.21.15.89:0/1568659864 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b94054da0 con 0x7f8b84004d90 2023-11-24T06:20:36.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.717+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8b9c0c82b0 con 0x7f8b84004d90 2023-11-24T06:20:36.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.718+0000 7f8b99ffb700 1 -- 172.21.15.89:0/1568659864 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8b940583c0 con 0x7f8b84004d90 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.719+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8b8806aed0 msgr2=0x7f8b8806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.719+0000 7f8ba35fa700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8b8806aed0 0x7f8b8806d390 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c1f4c40 tx=0x7f8b90009210 comp rx=0 tx=0).stop 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.719+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 msgr2=0x7f8b9c0caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.719+0000 7f8ba35fa700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b9c0caad0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8b94011aa0 tx=0x7f8b94011db0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.720+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 shutdown_connections 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.720+0000 7f8ba35fa700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8b8806aed0 0x7f8b8806d390 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.720+0000 7f8ba35fa700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b9c150f60 0x7f8b9c151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.720+0000 7f8ba35fa700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b84004d90 0x7f8b9c0caad0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.720+0000 7f8ba35fa700 1 --2- 172.21.15.89:0/1568659864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b9c0b4870 0x7f8b9c15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.720+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 >> 172.21.15.89:0/1568659864 conn(0x7f8b9c0b3780 msgr2=0x7f8b9c0c7240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:36.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.720+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 shutdown_connections 2023-11-24T06:20:36.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.720+0000 7f8ba35fa700 1 -- 172.21.15.89:0/1568659864 wait complete. 2023-11-24T06:20:36.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs subvol_grp1192 2023-11-24T06:20:36.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.854+0000 7f855a628700 1 Processor -- start 2023-11-24T06:20:36.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.854+0000 7f855a628700 1 -- start start 2023-11-24T06:20:36.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f855a628700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f855415e160 0x7f85540b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f855a628700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85540afc90 0x7f8554073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f855a628700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540b3c60 con 0x7f85540744d0 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f855a628700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85540b4500 con 0x7f85540afc90 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f855a628700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8554074940 con 0x7f855415e160 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f85537fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85540afc90 0x7f8554073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f85537fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85540afc90 0x7f8554073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38232/0 (socket says 172.21.15.89:38232) 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f85537fe700 1 -- 172.21.15.89:0/3810313810 learned_addr learned my addr 172.21.15.89:0/3810313810 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.855+0000 7f8553fff700 1 --2- 172.21.15.89:0/3810313810 >> v2:172.21.15.123:3300/0 conn(0x7f855415e160 0x7f85540b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f85537fe700 1 -- 172.21.15.89:0/3810313810 >> v2:172.21.15.123:3300/0 conn(0x7f855415e160 msgr2=0x7f85540b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f85537fe700 1 --2- 172.21.15.89:0/3810313810 >> v2:172.21.15.123:3300/0 conn(0x7f855415e160 0x7f85540b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f85537fe700 1 -- 172.21.15.89:0/3810313810 >> v1:172.21.15.123:6789/0 conn(0x7f85540744d0 legacy=0x7f855406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f85537fe700 1 -- 172.21.15.89:0/3810313810 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85540b58c0 con 0x7f85540afc90 2023-11-24T06:20:36.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f85537fe700 1 --2- 172.21.15.89:0/3810313810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540afc90 0x7f8554073f90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f855409efa0 tx=0x7f8544009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b47e698edde374 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f8552ffd700 1 -- 172.21.15.89:0/3810313810 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8544011830 con 0x7f85540afc90 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f8552ffd700 1 -- 172.21.15.89:0/3810313810 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8544011990 con 0x7f85540afc90 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f8552ffd700 1 -- 172.21.15.89:0/3810313810 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8544011d20 con 0x7f85540afc90 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f855a628700 1 -- 172.21.15.89:0/3810313810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540afc90 msgr2=0x7f8554073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f855a628700 1 --2- 172.21.15.89:0/3810313810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540afc90 0x7f8554073f90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f855409efa0 tx=0x7f8544009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f855a628700 1 -- 172.21.15.89:0/3810313810 shutdown_connections 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f855a628700 1 --2- 172.21.15.89:0/3810313810 >> v2:172.21.15.123:3300/0 conn(0x7f855415e160 0x7f85540b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f855a628700 1 --2- 172.21.15.89:0/3810313810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540afc90 0x7f8554073f90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f855a628700 1 -- 172.21.15.89:0/3810313810 >> 172.21.15.89:0/3810313810 conn(0x7f85540d0bf0 msgr2=0x7f85540d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f855a628700 1 -- 172.21.15.89:0/3810313810 shutdown_connections 2023-11-24T06:20:36.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.856+0000 7f855a628700 1 -- 172.21.15.89:0/3810313810 wait complete. 2023-11-24T06:20:36.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.858+0000 7f855a628700 1 Processor -- start 2023-11-24T06:20:36.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.858+0000 7f855a628700 1 -- start start 2023-11-24T06:20:36.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f855a628700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540744d0 0x7f8554149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f855a628700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540afc90 0x7f85540c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f855a628700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855415e160 0x7f85540c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f855a628700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85540ca3e0 con 0x7f85540744d0 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f855a628700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855414a650 con 0x7f85540afc90 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f855a628700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c6690 con 0x7f855415e160 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f8553fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540744d0 0x7f8554149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f8553fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540744d0 0x7f8554149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38236/0 (socket says 172.21.15.89:38236) 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f8553fff700 1 -- 172.21.15.89:0/1626882234 learned_addr learned my addr 172.21.15.89:0/1626882234 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f8553fff700 1 -- 172.21.15.89:0/1626882234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855415e160 msgr2=0x7f85540c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f8553fff700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855415e160 0x7f85540c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f8553fff700 1 -- 172.21.15.89:0/1626882234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540afc90 msgr2=0x7f85540c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f8553fff700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540afc90 0x7f85540c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:36.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.859+0000 7f8553fff700 1 -- 172.21.15.89:0/1626882234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8544009550 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.860+0000 7f8553fff700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540744d0 0x7f8554149db0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f854800e9b0 tx=0x7f854800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.860+0000 7f8550ff9700 1 -- 172.21.15.89:0/1626882234 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548009710 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.860+0000 7f8550ff9700 1 -- 172.21.15.89:0/1626882234 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854800c850 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.860+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85540c6970 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.860+0000 7f8550ff9700 1 -- 172.21.15.89:0/1626882234 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854800cbc0 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.860+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855414ab80 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.861+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85540c6b60 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.861+0000 7f8550ff9700 1 -- 172.21.15.89:0/1626882234 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f854800cd20 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.861+0000 7f8550ff9700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854006aed0 0x7f854006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.862+0000 7f8550ff9700 1 -- 172.21.15.89:0/1626882234 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85480896d0 con 0x7f85540744d0 2023-11-24T06:20:36.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.862+0000 7f85537fe700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854006aed0 0x7f854006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:36.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.862+0000 7f85537fe700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854006aed0 0x7f854006d390 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f85540d1520 tx=0x7f8544010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:36.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:36.866+0000 7f8550ff9700 1 -- 172.21.15.89:0/1626882234 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8548054780 con 0x7f85540744d0 2023-11-24T06:20:37.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.085+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "subvol_grp1192", "target": ["mon-mgr", ""]}) v1 -- 0x7f85540b3eb0 con 0x7f854006aed0 2023-11-24T06:20:37.092 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:20:37.089+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:20:37.092 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:20:37.089+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:20:37.092 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:20:37.089+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:20:37.092 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:20:37.089+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:20:37.092 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:20:37.089+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:20:37.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.103+0000 7f8550ff9700 1 -- 172.21.15.89:0/1626882234 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f854401f070 con 0x7f854006aed0 2023-11-24T06:20:37.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854006aed0 msgr2=0x7f854006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854006aed0 0x7f854006d390 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f85540d1520 tx=0x7f8544010000 comp rx=0 tx=0).stop 2023-11-24T06:20:37.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540744d0 msgr2=0x7f8554149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540744d0 0x7f8554149db0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f854800e9b0 tx=0x7f854800ed70 comp rx=0 tx=0).stop 2023-11-24T06:20:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 shutdown_connections 2023-11-24T06:20:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855415e160 0x7f85540c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85540744d0 0x7f8554149db0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f854006aed0 0x7f854006d390 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 --2- 172.21.15.89:0/1626882234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540afc90 0x7f85540c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 >> 172.21.15.89:0/1626882234 conn(0x7f85540d0bf0 msgr2=0x7f855406eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 shutdown_connections 2023-11-24T06:20:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.105+0000 7f855a628700 1 -- 172.21.15.89:0/1626882234 wait complete. 2023-11-24T06:20:37.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume info cephfs 2023-11-24T06:20:37.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.240+0000 7fbd62643700 1 Processor -- start 2023-11-24T06:20:37.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.240+0000 7fbd62643700 1 -- start start 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd62643700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd62643700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbd5c0746a0 0x7fbd5c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd62643700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c0b3c60 con 0x7fbd5c0afc90 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd62643700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c0b4500 con 0x7fbd5c0746a0 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd62643700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0b58c0 con 0x7fbd5c15e160 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd60be0700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbd5c0746a0 0x7fbd5c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd5bfff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd5b7fe700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fbd5c0afc90 0x7fbd5c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:45582/0 (socket says 172.21.15.89:45582) 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.241+0000 7fbd5b7fe700 1 -- 172.21.15.89:0/2631343578 learned_addr learned my addr 172.21.15.89:0/2631343578 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:37.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.242+0000 7fbd60be0700 1 -- 172.21.15.89:0/2631343578 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 msgr2=0x7fbd5c0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:37.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.242+0000 7fbd60be0700 1 --2- 172.21.15.89:0/2631343578 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c0b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.242+0000 7fbd60be0700 1 -- 172.21.15.89:0/2631343578 >> v1:172.21.15.123:6789/0 conn(0x7fbd5c0afc90 legacy=0x7fbd5c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.242+0000 7fbd60be0700 1 -- 172.21.15.89:0/2631343578 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c06bdc0 con 0x7fbd5c0746a0 2023-11-24T06:20:37.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.242+0000 7fbd60be0700 1 --2- 172.21.15.89:0/2631343578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 0x7fbd5c06d930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009d90 tx=0x7fbd5400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a67e02009b758d9d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:37.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.242+0000 7fbd5affd700 1 -- 172.21.15.89:0/2631343578 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5400e070 con 0x7fbd5c0746a0 2023-11-24T06:20:37.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.242+0000 7fbd5affd700 1 -- 172.21.15.89:0/2631343578 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5400bb20 con 0x7fbd5c0746a0 2023-11-24T06:20:37.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.242+0000 7fbd5affd700 1 -- 172.21.15.89:0/2631343578 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54012430 con 0x7fbd5c0746a0 2023-11-24T06:20:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.243+0000 7fbd62643700 1 -- 172.21.15.89:0/2631343578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 msgr2=0x7fbd5c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.243+0000 7fbd62643700 1 --2- 172.21.15.89:0/2631343578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 0x7fbd5c06d930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009d90 tx=0x7fbd5400b330 comp rx=0 tx=0).stop 2023-11-24T06:20:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.243+0000 7fbd62643700 1 -- 172.21.15.89:0/2631343578 shutdown_connections 2023-11-24T06:20:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.243+0000 7fbd62643700 1 --2- 172.21.15.89:0/2631343578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 0x7fbd5c06d930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.243+0000 7fbd62643700 1 --2- 172.21.15.89:0/2631343578 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.243+0000 7fbd62643700 1 -- 172.21.15.89:0/2631343578 >> 172.21.15.89:0/2631343578 conn(0x7fbd5c0d0bf0 msgr2=0x7fbd5c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.243+0000 7fbd62643700 1 -- 172.21.15.89:0/2631343578 shutdown_connections 2023-11-24T06:20:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.243+0000 7fbd62643700 1 -- 172.21.15.89:0/2631343578 wait complete. 2023-11-24T06:20:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.244+0000 7fbd62643700 1 Processor -- start 2023-11-24T06:20:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.244+0000 7fbd62643700 1 -- start start 2023-11-24T06:20:37.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd62643700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 0x7fbd5c158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd62643700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd5c0afc90 0x7fbd5c160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd62643700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd62643700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c160200 con 0x7fbd5c0afc90 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd62643700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c15f5f0 con 0x7fbd5c0746a0 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd62643700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c15f8d0 con 0x7fbd5c15e160 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd5c0afc90 0x7fbd5c160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd5c0afc90 0x7fbd5c160a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38262/0 (socket says 172.21.15.89:38262) 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5b7fe700 1 -- 172.21.15.89:0/871106234 learned_addr learned my addr 172.21.15.89:0/871106234 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:37.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd60be0700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5bfff700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 0x7fbd5c158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5b7fe700 1 -- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 msgr2=0x7fbd5c160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5b7fe700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5b7fe700 1 -- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 msgr2=0x7fbd5c158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5b7fe700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 0x7fbd5c158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5b7fe700 1 -- 172.21.15.89:0/871106234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd54009a70 con 0x7fbd5c0afc90 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd5bfff700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 0x7fbd5c158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.245+0000 7fbd60be0700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.246+0000 7fbd5b7fe700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd5c0afc90 0x7fbd5c160a60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c520 tx=0x7fbd4c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.246+0000 7fbd58ff9700 1 -- 172.21.15.89:0/871106234 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c005d70 con 0x7fbd5c0afc90 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.246+0000 7fbd58ff9700 1 -- 172.21.15.89:0/871106234 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c0070d0 con 0x7fbd5c0afc90 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.246+0000 7fbd58ff9700 1 -- 172.21.15.89:0/871106234 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c007440 con 0x7fbd5c0afc90 2023-11-24T06:20:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.246+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c15fbb0 con 0x7fbd5c0afc90 2023-11-24T06:20:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.246+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c158dd0 con 0x7fbd5c0afc90 2023-11-24T06:20:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.247+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c074e60 con 0x7fbd5c0afc90 2023-11-24T06:20:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.247+0000 7fbd58ff9700 1 -- 172.21.15.89:0/871106234 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbd4c0075a0 con 0x7fbd5c0afc90 2023-11-24T06:20:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.248+0000 7fbd58ff9700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbd4806aed0 0x7fbd4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.248+0000 7fbd58ff9700 1 -- 172.21.15.89:0/871106234 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd4c0889f0 con 0x7fbd5c0afc90 2023-11-24T06:20:37.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.248+0000 7fbd5bfff700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbd4806aed0 0x7fbd4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.248+0000 7fbd5bfff700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbd4806aed0 0x7fbd4806d390 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009b90 tx=0x7fbd50008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:37.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.252+0000 7fbd58ff9700 1 -- 172.21.15.89:0/871106234 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd4c053b40 con 0x7fbd5c0afc90 2023-11-24T06:20:37.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.470+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs volume info", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd5c0b0110 con 0x7fbd4806aed0 2023-11-24T06:20:37.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.472+0000 7fbd58ff9700 1 -- 172.21.15.89:0/871106234 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+530 (secure 0 0 0) 0x7fbd500097e0 con 0x7fbd4806aed0 2023-11-24T06:20:37.475 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:20:37.475 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_addrs": [ 2023-11-24T06:20:37.475 INFO:teuthology.orchestra.run.smithi089.stdout: "172.21.15.89:6789", 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "172.21.15.123:6789", 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "172.21.15.123:6790" 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "pending_subvolume_deletions": 0, 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": { 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "data": [ 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "avail": 367104032768, 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs_data", 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "used": 0 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2023-11-24T06:20:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: "metadata": [ 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: "avail": 367104032768, 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs_metadata", 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: "used": 65536 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout: "used_size": 0 2023-11-24T06:20:37.477 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:20:37.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbd4806aed0 msgr2=0x7fbd4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbd4806aed0 0x7fbd4806d390 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009b90 tx=0x7fbd50008040 comp rx=0 tx=0).stop 2023-11-24T06:20:37.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd5c0afc90 msgr2=0x7fbd5c160a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd5c0afc90 0x7fbd5c160a60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c520 tx=0x7fbd4c00c8e0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 shutdown_connections 2023-11-24T06:20:37.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd5c15e160 0x7fbd5c160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fbd4806aed0 0x7fbd4806d390 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd5c0746a0 0x7fbd5c158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 --2- 172.21.15.89:0/871106234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd5c0afc90 0x7fbd5c160a60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 >> 172.21.15.89:0/871106234 conn(0x7fbd5c0d0bf0 msgr2=0x7fbd5c0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:37.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 shutdown_connections 2023-11-24T06:20:37.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.474+0000 7fbd62643700 1 -- 172.21.15.89:0/871106234 wait complete. 2023-11-24T06:20:37.489 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T06:20:37.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:20:37.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:20:37.573 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T06:20:37.573 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:37.574 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:37.591 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:37.591 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:37.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:37.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.808+0000 7fdcf0097700 1 Processor -- start 2023-11-24T06:20:37.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.808+0000 7fdcf0097700 1 -- start start 2023-11-24T06:20:37.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcf0097700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce815bdd0 0x7fdce81604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcf0097700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdce80b3370 0x7fdce80b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcf0097700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce80b2030 con 0x7fdce80d1c70 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcf0097700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce80b2f50 con 0x7fdce80b3370 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcf0097700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80b22d0 con 0x7fdce815bdd0 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdced632700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdce80b3370 0x7fdce80b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcee634700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdce80d1c70 0x7fdce80ce6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41776/0 (socket says 172.21.15.89:41776) 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcee634700 1 -- 172.21.15.89:0/3984233427 learned_addr learned my addr 172.21.15.89:0/3984233427 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcede33700 1 --2- 172.21.15.89:0/3984233427 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce815bdd0 0x7fdce81604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdced632700 1 -- 172.21.15.89:0/3984233427 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce815bdd0 msgr2=0x7fdce81604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdced632700 1 --2- 172.21.15.89:0/3984233427 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce815bdd0 0x7fdce81604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdced632700 1 -- 172.21.15.89:0/3984233427 >> v1:172.21.15.89:6789/0 conn(0x7fdce80d1c70 legacy=0x7fdce80ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdced632700 1 -- 172.21.15.89:0/3984233427 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce80b25b0 con 0x7fdce80b3370 2023-11-24T06:20:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdcece31700 1 -- 172.21.15.89:0/3984233427 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990296503 0 0) 0x7fdce80b2030 con 0x7fdce80d1c70 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.809+0000 7fdced632700 1 --2- 172.21.15.89:0/3984233427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce80b3370 0x7fdce80b1af0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009840 tx=0x7fdcd8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227e2ec70a6a0134 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcece31700 1 -- 172.21.15.89:0/3984233427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8011830 con 0x7fdce80b3370 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcece31700 1 -- 172.21.15.89:0/3984233427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd8011990 con 0x7fdce80b3370 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcece31700 1 -- 172.21.15.89:0/3984233427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8011d20 con 0x7fdce80b3370 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcf0097700 1 -- 172.21.15.89:0/3984233427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce80b3370 msgr2=0x7fdce80b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcf0097700 1 --2- 172.21.15.89:0/3984233427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce80b3370 0x7fdce80b1af0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009840 tx=0x7fdcd8009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcf0097700 1 -- 172.21.15.89:0/3984233427 shutdown_connections 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcf0097700 1 --2- 172.21.15.89:0/3984233427 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce815bdd0 0x7fdce81604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcf0097700 1 --2- 172.21.15.89:0/3984233427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce80b3370 0x7fdce80b1af0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcf0097700 1 -- 172.21.15.89:0/3984233427 >> 172.21.15.89:0/3984233427 conn(0x7fdce80731a0 msgr2=0x7fdce809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcf0097700 1 -- 172.21.15.89:0/3984233427 shutdown_connections 2023-11-24T06:20:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.810+0000 7fdcf0097700 1 -- 172.21.15.89:0/3984233427 wait complete. 2023-11-24T06:20:37.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.811+0000 7fdcf0097700 1 Processor -- start 2023-11-24T06:20:37.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.811+0000 7fdcf0097700 1 -- start start 2023-11-24T06:20:37.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcf0097700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80b3370 0x7fdce814fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcf0097700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80d1c70 0x7fdce815c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcf0097700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce815bdd0 0x7fdce815c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcf0097700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce81503e0 con 0x7fdce815bdd0 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcf0097700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80b51e0 con 0x7fdce80b3370 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcf0097700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80b48d0 con 0x7fdce80d1c70 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcee634700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce815bdd0 0x7fdce815c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcee634700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce815bdd0 0x7fdce815c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38278/0 (socket says 172.21.15.89:38278) 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcee634700 1 -- 172.21.15.89:0/518171774 learned_addr learned my addr 172.21.15.89:0/518171774 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdced632700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80d1c70 0x7fdce815c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.812+0000 7fdcede33700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80b3370 0x7fdce814fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcee634700 1 -- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80d1c70 msgr2=0x7fdce815c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcee634700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80d1c70 0x7fdce815c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcee634700 1 -- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80b3370 msgr2=0x7fdce814fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcee634700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80b3370 0x7fdce814fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcee634700 1 -- 172.21.15.89:0/518171774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80094a0 con 0x7fdce815bdd0 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdced632700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80d1c70 0x7fdce815c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcede33700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80b3370 0x7fdce814fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcee634700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce815bdd0 0x7fdce815c9f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdce0002230 tx=0x7fdce000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcde7fc700 1 -- 172.21.15.89:0/518171774 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce000ac20 con 0x7fdce815bdd0 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcde7fc700 1 -- 172.21.15.89:0/518171774 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce000ada0 con 0x7fdce815bdd0 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcde7fc700 1 -- 172.21.15.89:0/518171774 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0007310 con 0x7fdce815bdd0 2023-11-24T06:20:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce80cb640 con 0x7fdce815bdd0 2023-11-24T06:20:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.813+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce80b4e70 con 0x7fdce815bdd0 2023-11-24T06:20:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.814+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce8067f60 con 0x7fdce815bdd0 2023-11-24T06:20:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.814+0000 7fdcde7fc700 1 -- 172.21.15.89:0/518171774 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdce0007470 con 0x7fdce815bdd0 2023-11-24T06:20:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.815+0000 7fdcde7fc700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdccc06aed0 0x7fdccc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.815+0000 7fdcde7fc700 1 -- 172.21.15.89:0/518171774 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdce0089170 con 0x7fdce815bdd0 2023-11-24T06:20:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.815+0000 7fdcede33700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdccc06aed0 0x7fdccc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.815+0000 7fdcede33700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdccc06aed0 0x7fdccc06d390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdce4007a50 tx=0x7fdce4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:37.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:37.819+0000 7fdcde7fc700 1 -- 172.21.15.89:0/518171774 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdce0054220 con 0x7fdce815bdd0 2023-11-24T06:20:38.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.009+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdce80b5320 con 0x7fdce815bdd0 2023-11-24T06:20:38.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.011+0000 7fdcde7fc700 1 -- 172.21.15.89:0/518171774 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16529 (secure 0 0 0) 0x7fdce0057840 con 0x7fdce815bdd0 2023-11-24T06:20:38.014 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:38.014 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":53,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:23.644194+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:20.413276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:20.978712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:38.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdccc06aed0 msgr2=0x7fdccc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdccc06aed0 0x7fdccc06d390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdce4007a50 tx=0x7fdce4008040 comp rx=0 tx=0).stop 2023-11-24T06:20:38.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce815bdd0 msgr2=0x7fdce815c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce815bdd0 0x7fdce815c9f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdce0002230 tx=0x7fdce000a060 comp rx=0 tx=0).stop 2023-11-24T06:20:38.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 shutdown_connections 2023-11-24T06:20:38.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdce815bdd0 0x7fdce815c9f0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdccc06aed0 0x7fdccc06d390 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80b3370 0x7fdce814fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 --2- 172.21.15.89:0/518171774 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80d1c70 0x7fdce815c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 >> 172.21.15.89:0/518171774 conn(0x7fdce80731a0 msgr2=0x7fdce80afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 shutdown_connections 2023-11-24T06:20:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.014+0000 7fdcf0097700 1 -- 172.21.15.89:0/518171774 wait complete. 2023-11-24T06:20:38.030 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:20:20.413276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:20:20.978712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:38.031 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:38.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.151+0000 7f7b3a8f9700 1 Processor -- start 2023-11-24T06:20:38.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.151+0000 7f7b3a8f9700 1 -- start start 2023-11-24T06:20:38.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b3a8f9700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b3415bf50 0x7f7b340ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b3a8f9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b340ce830 0x7f7b340cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b3a8f9700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b340d0140 con 0x7f7b340cf1f0 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b3a8f9700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b3415c330 con 0x7f7b3415bf50 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b3a8f9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b340d03e0 con 0x7f7b340ce830 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b33fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b3415bf50 0x7f7b340ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b33fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b3415bf50 0x7f7b340ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38282/0 (socket says 172.21.15.89:38282) 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b33fff700 1 -- 172.21.15.89:0/3782197317 learned_addr learned my addr 172.21.15.89:0/3782197317 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b33fff700 1 -- 172.21.15.89:0/3782197317 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b340ce830 msgr2=0x7f7b340cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b337fe700 1 --2- 172.21.15.89:0/3782197317 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b340ce830 0x7f7b340cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b33fff700 1 --2- 172.21.15.89:0/3782197317 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b340ce830 0x7f7b340cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b33fff700 1 -- 172.21.15.89:0/3782197317 >> v1:172.21.15.123:6789/0 conn(0x7f7b340cf1f0 legacy=0x7f7b340c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:38.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b33fff700 1 -- 172.21.15.89:0/3782197317 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b340d06c0 con 0x7f7b3415bf50 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b337fe700 1 --2- 172.21.15.89:0/3782197317 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b340ce830 0x7f7b340cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b33fff700 1 --2- 172.21.15.89:0/3782197317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b3415bf50 0x7f7b340ce2f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7b280098f0 tx=0x7f7b28009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7e9e31f1900f9b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b32ffd700 1 -- 172.21.15.89:0/3782197317 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28004230 con 0x7f7b3415bf50 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b32ffd700 1 -- 172.21.15.89:0/3782197317 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b28004390 con 0x7f7b3415bf50 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.152+0000 7f7b32ffd700 1 -- 172.21.15.89:0/3782197317 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b280036a0 con 0x7f7b3415bf50 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.153+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/3782197317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b3415bf50 msgr2=0x7f7b340ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.153+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/3782197317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b3415bf50 0x7f7b340ce2f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7b280098f0 tx=0x7f7b28009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.153+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/3782197317 shutdown_connections 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.153+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/3782197317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b3415bf50 0x7f7b340ce2f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.153+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/3782197317 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b340ce830 0x7f7b340cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.153+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/3782197317 >> 172.21.15.89:0/3782197317 conn(0x7f7b340b3780 msgr2=0x7f7b340b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:38.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.153+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/3782197317 shutdown_connections 2023-11-24T06:20:38.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.153+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/3782197317 wait complete. 2023-11-24T06:20:38.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.154+0000 7f7b3a8f9700 1 Processor -- start 2023-11-24T06:20:38.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.154+0000 7f7b3a8f9700 1 -- start start 2023-11-24T06:20:38.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b3a8f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b340ce830 0x7f7b3414bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b3a8f9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b340cf1f0 0x7f7b34070240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b3a8f9700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b3415bf50 0x7f7b34070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b3a8f9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3409bb70 con 0x7f7b340ce830 2023-11-24T06:20:38.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b3a8f9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3409e410 con 0x7f7b340cf1f0 2023-11-24T06:20:38.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b3a8f9700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3409bd80 con 0x7f7b3415bf50 2023-11-24T06:20:38.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b33fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b340ce830 0x7f7b3414bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b33fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b340ce830 0x7f7b3414bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38288/0 (socket says 172.21.15.89:38288) 2023-11-24T06:20:38.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b33fff700 1 -- 172.21.15.89:0/1847424302 learned_addr learned my addr 172.21.15.89:0/1847424302 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:38.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b337fe700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b340cf1f0 0x7f7b34070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b38e96700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b3415bf50 0x7f7b34070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b33fff700 1 -- 172.21.15.89:0/1847424302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b3415bf50 msgr2=0x7f7b34070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.155+0000 7f7b33fff700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b3415bf50 0x7f7b34070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b33fff700 1 -- 172.21.15.89:0/1847424302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b340cf1f0 msgr2=0x7f7b34070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b33fff700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b340cf1f0 0x7f7b34070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b33fff700 1 -- 172.21.15.89:0/1847424302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b28009550 con 0x7f7b340ce830 2023-11-24T06:20:38.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b33fff700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b340ce830 0x7f7b3414bec0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009130 tx=0x7f7b28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b30ff9700 1 -- 172.21.15.89:0/1847424302 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2800b860 con 0x7f7b340ce830 2023-11-24T06:20:38.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b30ff9700 1 -- 172.21.15.89:0/1847424302 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2800b9c0 con 0x7f7b340ce830 2023-11-24T06:20:38.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b30ff9700 1 -- 172.21.15.89:0/1847424302 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2800bd70 con 0x7f7b340ce830 2023-11-24T06:20:38.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3409c060 con 0x7f7b340ce830 2023-11-24T06:20:38.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.156+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3409c620 con 0x7f7b340ce830 2023-11-24T06:20:38.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.157+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3409c7b0 con 0x7f7b340ce830 2023-11-24T06:20:38.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.157+0000 7f7b30ff9700 1 -- 172.21.15.89:0/1847424302 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7b2801b440 con 0x7f7b340ce830 2023-11-24T06:20:38.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.158+0000 7f7b30ff9700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7b2006aed0 0x7f7b2006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.158+0000 7f7b30ff9700 1 -- 172.21.15.89:0/1847424302 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b2808af80 con 0x7f7b340ce830 2023-11-24T06:20:38.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.158+0000 7f7b337fe700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7b2006aed0 0x7f7b2006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.158+0000 7f7b337fe700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7b2006aed0 0x7f7b2006d390 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7b340c53a0 tx=0x7f7b24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.162+0000 7f7b30ff9700 1 -- 172.21.15.89:0/1847424302 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b280bde90 con 0x7f7b340ce830 2023-11-24T06:20:38.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.352+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b34068f90 con 0x7f7b340ce830 2023-11-24T06:20:38.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.352+0000 7f7b30ff9700 1 -- 172.21.15.89:0/1847424302 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b280560c0 con 0x7f7b340ce830 2023-11-24T06:20:38.356 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:38.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7b2006aed0 msgr2=0x7f7b2006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7b2006aed0 0x7f7b2006d390 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7b340c53a0 tx=0x7f7b24008040 comp rx=0 tx=0).stop 2023-11-24T06:20:38.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b340ce830 msgr2=0x7f7b3414bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b340ce830 0x7f7b3414bec0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009130 tx=0x7f7b28004750 comp rx=0 tx=0).stop 2023-11-24T06:20:38.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 shutdown_connections 2023-11-24T06:20:38.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b3415bf50 0x7f7b34070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b340ce830 0x7f7b3414bec0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7b2006aed0 0x7f7b2006d390 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 --2- 172.21.15.89:0/1847424302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b340cf1f0 0x7f7b34070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.354+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 >> 172.21.15.89:0/1847424302 conn(0x7f7b340b3780 msgr2=0x7f7b340c5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:38.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.355+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 shutdown_connections 2023-11-24T06:20:38.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.355+0000 7f7b3a8f9700 1 -- 172.21.15.89:0/1847424302 wait complete. 2023-11-24T06:20:38.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:20:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.491+0000 7fd71b171700 1 Processor -- start 2023-11-24T06:20:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.491+0000 7fd71b171700 1 -- start start 2023-11-24T06:20:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd71b171700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd71415ada0 0x7fd71406f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd71b171700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7140afb40 0x7fd7140cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd71b171700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7140cccd0 con 0x7fd7140cfa80 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd71b171700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7140cd2e0 con 0x7fd71415ada0 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd71b171700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7140cd5c0 con 0x7fd7140afb40 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd718f0d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd71415ada0 0x7fd71406f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd71970e700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd7140cfa80 0x7fd71406d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41802/0 (socket says 172.21.15.89:41802) 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd71970e700 1 -- 172.21.15.89:0/2333080530 learned_addr learned my addr 172.21.15.89:0/2333080530 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd718f0d700 1 -- 172.21.15.89:0/2333080530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7140afb40 msgr2=0x7fd7140cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd713fff700 1 --2- 172.21.15.89:0/2333080530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7140afb40 0x7fd7140cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd7137fe700 1 -- 172.21.15.89:0/2333080530 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722697746 0 0) 0x7fd7140cccd0 con 0x7fd7140cfa80 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd718f0d700 1 --2- 172.21.15.89:0/2333080530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7140afb40 0x7fd7140cf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd718f0d700 1 -- 172.21.15.89:0/2333080530 >> v1:172.21.15.89:6789/0 conn(0x7fd7140cfa80 legacy=0x7fd71406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd718f0d700 1 -- 172.21.15.89:0/2333080530 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71406fd50 con 0x7fd71415ada0 2023-11-24T06:20:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd718f0d700 1 --2- 172.21.15.89:0/2333080530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd71415ada0 0x7fd71406f810 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd708009840 tx=0x7fd708009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d141cb0aa7ee77ab server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.492+0000 7fd7137fe700 1 -- 172.21.15.89:0/2333080530 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708011830 con 0x7fd71415ada0 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd7137fe700 1 -- 172.21.15.89:0/2333080530 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd708011990 con 0x7fd71415ada0 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd7137fe700 1 -- 172.21.15.89:0/2333080530 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708011d20 con 0x7fd71415ada0 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd71b171700 1 -- 172.21.15.89:0/2333080530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd71415ada0 msgr2=0x7fd71406f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd71b171700 1 --2- 172.21.15.89:0/2333080530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd71415ada0 0x7fd71406f810 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd708009840 tx=0x7fd708009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd71b171700 1 -- 172.21.15.89:0/2333080530 shutdown_connections 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd71b171700 1 --2- 172.21.15.89:0/2333080530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd71415ada0 0x7fd71406f810 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd71b171700 1 --2- 172.21.15.89:0/2333080530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7140afb40 0x7fd7140cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd71b171700 1 -- 172.21.15.89:0/2333080530 >> 172.21.15.89:0/2333080530 conn(0x7fd7140b2050 msgr2=0x7fd7140b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd71b171700 1 -- 172.21.15.89:0/2333080530 shutdown_connections 2023-11-24T06:20:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.493+0000 7fd71b171700 1 -- 172.21.15.89:0/2333080530 wait complete. 2023-11-24T06:20:38.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.494+0000 7fd71b171700 1 Processor -- start 2023-11-24T06:20:38.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.494+0000 7fd71b171700 1 -- start start 2023-11-24T06:20:38.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd71b171700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7140afb40 0x7fd714140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd71b171700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7140cfa80 0x7fd714159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd71b171700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71415ada0 0x7fd714159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd71b171700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd714141160 con 0x7fd7140cfa80 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd71b171700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd714141550 con 0x7fd7140afb40 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd71b171700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71415a250 con 0x7fd71415ada0 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd713fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7140cfa80 0x7fd714159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd713fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7140cfa80 0x7fd714159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38304/0 (socket says 172.21.15.89:38304) 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd713fff700 1 -- 172.21.15.89:0/2766757021 learned_addr learned my addr 172.21.15.89:0/2766757021 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd713fff700 1 -- 172.21.15.89:0/2766757021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71415ada0 msgr2=0x7fd714159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd713fff700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71415ada0 0x7fd714159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd713fff700 1 -- 172.21.15.89:0/2766757021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7140afb40 msgr2=0x7fd714140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.495+0000 7fd718f0d700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7140afb40 0x7fd714140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.496+0000 7fd713fff700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7140afb40 0x7fd714140c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.496+0000 7fd713fff700 1 -- 172.21.15.89:0/2766757021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7080094a0 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.496+0000 7fd713fff700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7140cfa80 0x7fd714159700 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd70000c7f0 tx=0x7fd70000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.496+0000 7fd7117fa700 1 -- 172.21.15.89:0/2766757021 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd700011800 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.496+0000 7fd7117fa700 1 -- 172.21.15.89:0/2766757021 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd700011960 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.496+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd714144330 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.496+0000 7fd7117fa700 1 -- 172.21.15.89:0/2766757021 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd700011cf0 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.496+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd714141ad0 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.497+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd714068d90 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.497+0000 7fd7117fa700 1 -- 172.21.15.89:0/2766757021 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd700018360 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.498+0000 7fd7117fa700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd6f806aed0 0x7fd6f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.498+0000 7fd7117fa700 1 -- 172.21.15.89:0/2766757021 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd700089540 con 0x7fd7140cfa80 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.498+0000 7fd718f0d700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd6f806aed0 0x7fd6f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.498+0000 7fd718f0d700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd6f806aed0 0x7fd6f806d390 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd708000c00 tx=0x7fd708008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.502+0000 7fd7117fa700 1 -- 172.21.15.89:0/2766757021 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd700054600 con 0x7fd7140cfa80 2023-11-24T06:20:38.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.697+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd714068fb0 con 0x7fd7140cfa80 2023-11-24T06:20:38.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.697+0000 7fd7117fa700 1 -- 172.21.15.89:0/2766757021 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd700057c20 con 0x7fd7140cfa80 2023-11-24T06:20:38.700 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:20:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd6f806aed0 msgr2=0x7fd6f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd6f806aed0 0x7fd6f806d390 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd708000c00 tx=0x7fd708008030 comp rx=0 tx=0).stop 2023-11-24T06:20:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7140cfa80 msgr2=0x7fd714159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7140cfa80 0x7fd714159700 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd70000c7f0 tx=0x7fd70000cbb0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 shutdown_connections 2023-11-24T06:20:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71415ada0 0x7fd714159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd6f806aed0 0x7fd6f806d390 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7140afb40 0x7fd714140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 --2- 172.21.15.89:0/2766757021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7140cfa80 0x7fd714159700 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 >> 172.21.15.89:0/2766757021 conn(0x7fd7140b2050 msgr2=0x7fd7140b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 shutdown_connections 2023-11-24T06:20:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.699+0000 7fd71b171700 1 -- 172.21.15.89:0/2766757021 wait complete. 2023-11-24T06:20:38.714 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:20:38.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.839+0000 7f72cfe19700 1 Processor -- start 2023-11-24T06:20:38.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.839+0000 7f72cfe19700 1 -- start start 2023-11-24T06:20:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72cfe19700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f72c815e160 0x7f72c80b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72cfe19700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80afc90 0x7f72c8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72cfe19700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c80b3c60 con 0x7f72c80744d0 2023-11-24T06:20:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72cfe19700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72c80b4500 con 0x7f72c815e160 2023-11-24T06:20:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72cfe19700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c8074960 con 0x7f72c80afc90 2023-11-24T06:20:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72cdbb5700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f72c815e160 0x7f72c80b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72cd3b4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80afc90 0x7f72c8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72ce3b6700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f72c80744d0 0x7f72c806d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:45638/0 (socket says 172.21.15.89:45638) 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.840+0000 7f72ce3b6700 1 -- 172.21.15.89:0/1589595208 learned_addr learned my addr 172.21.15.89:0/1589595208 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72cd3b4700 1 -- 172.21.15.89:0/1589595208 >> v2:172.21.15.123:3300/0 conn(0x7f72c815e160 msgr2=0x7f72c80b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72cd3b4700 1 --2- 172.21.15.89:0/1589595208 >> v2:172.21.15.123:3300/0 conn(0x7f72c815e160 0x7f72c80b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72cd3b4700 1 -- 172.21.15.89:0/1589595208 >> v1:172.21.15.123:6789/0 conn(0x7f72c80744d0 legacy=0x7f72c806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72cd3b4700 1 -- 172.21.15.89:0/1589595208 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c80b58c0 con 0x7f72c80afc90 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72cdbb5700 1 --2- 172.21.15.89:0/1589595208 >> v2:172.21.15.123:3300/0 conn(0x7f72c815e160 0x7f72c80b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72cd3b4700 1 --2- 172.21.15.89:0/1589595208 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80afc90 0x7f72c8073f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009840 tx=0x7f72c4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6efc752cc4f1a019 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72ccbb3700 1 -- 172.21.15.89:0/1589595208 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4011830 con 0x7f72c80afc90 2023-11-24T06:20:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72ccbb3700 1 -- 172.21.15.89:0/1589595208 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c4011990 con 0x7f72c80afc90 2023-11-24T06:20:38.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.841+0000 7f72ccbb3700 1 -- 172.21.15.89:0/1589595208 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4011d20 con 0x7f72c80afc90 2023-11-24T06:20:38.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.842+0000 7f72cfe19700 1 -- 172.21.15.89:0/1589595208 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80afc90 msgr2=0x7f72c8073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.842+0000 7f72cfe19700 1 --2- 172.21.15.89:0/1589595208 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80afc90 0x7f72c8073f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009840 tx=0x7f72c4009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:38.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.842+0000 7f72cfe19700 1 -- 172.21.15.89:0/1589595208 shutdown_connections 2023-11-24T06:20:38.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.842+0000 7f72cfe19700 1 --2- 172.21.15.89:0/1589595208 >> v2:172.21.15.123:3300/0 conn(0x7f72c815e160 0x7f72c80b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.842+0000 7f72cfe19700 1 --2- 172.21.15.89:0/1589595208 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80afc90 0x7f72c8073f90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.842+0000 7f72cfe19700 1 -- 172.21.15.89:0/1589595208 >> 172.21.15.89:0/1589595208 conn(0x7f72c80d0bf0 msgr2=0x7f72c80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:38.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.842+0000 7f72cfe19700 1 -- 172.21.15.89:0/1589595208 shutdown_connections 2023-11-24T06:20:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.842+0000 7f72cfe19700 1 -- 172.21.15.89:0/1589595208 wait complete. 2023-11-24T06:20:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.843+0000 7f72cfe19700 1 Processor -- start 2023-11-24T06:20:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.843+0000 7f72cfe19700 1 -- start start 2023-11-24T06:20:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cfe19700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80744d0 0x7f72c80c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cfe19700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c80afc90 0x7f72c80c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cfe19700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72c815e160 0x7f72c80c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cfe19700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c80c7660 con 0x7f72c815e160 2023-11-24T06:20:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cfe19700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c80c8830 con 0x7f72c80afc90 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cfe19700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c80cc220 con 0x7f72c80744d0 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72c815e160 0x7f72c80c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72c815e160 0x7f72c80c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38320/0 (socket says 172.21.15.89:38320) 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 -- 172.21.15.89:0/4222899951 learned_addr learned my addr 172.21.15.89:0/4222899951 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cdbb5700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80744d0 0x7f72c80c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cd3b4700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c80afc90 0x7f72c80c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 -- 172.21.15.89:0/4222899951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80744d0 msgr2=0x7f72c80c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80744d0 0x7f72c80c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 -- 172.21.15.89:0/4222899951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c80afc90 msgr2=0x7f72c80c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c80afc90 0x7f72c80c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 -- 172.21.15.89:0/4222899951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b80094a0 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72ce3b6700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72c815e160 0x7f72c80c9b00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f72c000cbf0 tx=0x7f72c000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72be7fc700 1 -- 172.21.15.89:0/4222899951 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c000aa60 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72be7fc700 1 -- 172.21.15.89:0/4222899951 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c000abc0 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c40094a0 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.844+0000 7f72be7fc700 1 -- 172.21.15.89:0/4222899951 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0012400 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.845+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c80c9020 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.845+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c815d360 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.846+0000 7f72be7fc700 1 -- 172.21.15.89:0/4222899951 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f72c001b3b0 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.846+0000 7f72be7fc700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f72ac06aed0 0x7f72ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.846+0000 7f72be7fc700 1 -- 172.21.15.89:0/4222899951 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72c0089080 con 0x7f72c815e160 2023-11-24T06:20:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.846+0000 7f72cdbb5700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f72ac06aed0 0x7f72ac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:38.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.847+0000 7f72cdbb5700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f72ac06aed0 0x7f72ac06d390 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009cd0 tx=0x7f72b8009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:38.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:38.851+0000 7f72be7fc700 1 -- 172.21.15.89:0/4222899951 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72c00541e0 con 0x7f72c815e160 2023-11-24T06:20:39.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.027+0000 7f72be7fc700 1 -- 172.21.15.89:0/4222899951 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f72c00545b0 con 0x7f72c815e160 2023-11-24T06:20:39.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.043+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f72c8068fb0 con 0x7f72c815e160 2023-11-24T06:20:39.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.043+0000 7f72be7fc700 1 -- 172.21.15.89:0/4222899951 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7f72c00737c0 con 0x7f72c815e160 2023-11-24T06:20:39.046 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f72ac06aed0 msgr2=0x7f72ac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f72ac06aed0 0x7f72ac06d390 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009cd0 tx=0x7f72b8009210 comp rx=0 tx=0).stop 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72c815e160 msgr2=0x7f72c80c9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72c815e160 0x7f72c80c9b00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f72c000cbf0 tx=0x7f72c000cf00 comp rx=0 tx=0).stop 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 shutdown_connections 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72c815e160 0x7f72c80c9b00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c80744d0 0x7f72c80c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f72ac06aed0 0x7f72ac06d390 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 --2- 172.21.15.89:0/4222899951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c80afc90 0x7f72c80c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.045+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 >> 172.21.15.89:0/4222899951 conn(0x7f72c80d0bf0 msgr2=0x7f72c8071bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.046+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 shutdown_connections 2023-11-24T06:20:39.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.046+0000 7f72cfe19700 1 -- 172.21.15.89:0/4222899951 wait complete. 2023-11-24T06:20:39.060 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:20:39.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.180+0000 7fc42e27a700 1 Processor -- start 2023-11-24T06:20:39.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.180+0000 7fc42e27a700 1 -- start start 2023-11-24T06:20:39.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.181+0000 7fc42e27a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc42815e080 0x7fc4280b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.181+0000 7fc42e27a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc42806c710 0x7fc4280afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.181+0000 7fc42e27a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280ae430 con 0x7fc4280af4d0 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.181+0000 7fc42e27a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4280af0b0 con 0x7fc42806c710 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.181+0000 7fc42e27a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280b05a0 con 0x7fc42815e080 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.181+0000 7fc4277fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4280af4d0 0x7fc42806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41828/0 (socket says 172.21.15.89:41828) 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.181+0000 7fc4277fe700 1 -- 172.21.15.89:0/3704823229 learned_addr learned my addr 172.21.15.89:0/3704823229 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc427fff700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc42815e080 0x7fc4280b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc42c817700 1 --2- 172.21.15.89:0/3704823229 >> v2:172.21.15.123:3300/0 conn(0x7fc42806c710 0x7fc4280afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031536865 0 0) 0x7fc4280ae430 con 0x7fc4280af4d0 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4100035b0 con 0x7fc4280af4d0 2023-11-24T06:20:39.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898824612 0 0) 0x7fc4100035b0 con 0x7fc4280af4d0 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc42815e080 msgr2=0x7fc4280b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc42815e080 0x7fc4280b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 >> v2:172.21.15.123:3300/0 conn(0x7fc42806c710 msgr2=0x7fc4280afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 --2- 172.21.15.89:0/3704823229 >> v2:172.21.15.123:3300/0 conn(0x7fc42806c710 0x7fc4280afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4280ae710 con 0x7fc4280af4d0 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fc418001670 con 0x7fc4280af4d0 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 >> v1:172.21.15.89:6789/0 conn(0x7fc4280af4d0 legacy=0x7fc42806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc410003a30 0x7fc410003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc410004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc42815e080 0x7fc410005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280af0b0 con 0x7fc410004850 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280b05a0 con 0x7fc410003a30 2023-11-24T06:20:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41000ba10 con 0x7fc42815e080 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc4277fe700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc410004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc427fff700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc410003a30 0x7fc410003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc42c817700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc42815e080 0x7fc410005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc4277fe700 1 -- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc42815e080 msgr2=0x7fc410005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc4277fe700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc42815e080 0x7fc410005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc4277fe700 1 -- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc410003a30 msgr2=0x7fc410003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc4277fe700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc410003a30 0x7fc410003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc4277fe700 1 -- 172.21.15.89:0/3704823229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4100035b0 con 0x7fc410004850 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.182+0000 7fc4277fe700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc410004cd0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc418001050 tx=0x7fc410005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2437cbb508542c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41800cd50 con 0x7fc410004850 2023-11-24T06:20:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418005960 con 0x7fc410004850 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc426ffd700 1 -- 172.21.15.89:0/3704823229 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418005d10 con 0x7fc410004850 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc42e27a700 1 -- 172.21.15.89:0/3704823229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 msgr2=0x7fc410004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc42e27a700 1 --2- 172.21.15.89:0/3704823229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc410004cd0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc418001050 tx=0x7fc410005880 comp rx=0 tx=0).stop 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc42c817700 1 -- 172.21.15.89:0/3704823229 reap_dead start 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc42e27a700 1 -- 172.21.15.89:0/3704823229 shutdown_connections 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc42e27a700 1 -- 172.21.15.89:0/3704823229 >> 172.21.15.89:0/3704823229 conn(0x7fc428071b90 msgr2=0x7fc428071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc42e27a700 1 -- 172.21.15.89:0/3704823229 shutdown_connections 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.183+0000 7fc42e27a700 1 -- 172.21.15.89:0/3704823229 wait complete. 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.184+0000 7fc42e27a700 1 Processor -- start 2023-11-24T06:20:39.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.185+0000 7fc42e27a700 1 -- start start 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.185+0000 7fc42e27a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc428159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.185+0000 7fc42e27a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc428154bd0 0x7fc428153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.185+0000 7fc42e27a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc428160a10 0x7fc428160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.185+0000 7fc42e27a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4281613d0 con 0x7fc410004850 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.185+0000 7fc42e27a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc428155050 con 0x7fc428154bd0 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.185+0000 7fc42e27a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc428155310 con 0x7fc428160a10 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.185+0000 7fc427fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc428159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc427fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc428159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38336/0 (socket says 172.21.15.89:38336) 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc427fff700 1 -- 172.21.15.89:0/2866785700 learned_addr learned my addr 172.21.15.89:0/2866785700 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc4277fe700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc428154bd0 0x7fc428153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc42c817700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc428160a10 0x7fc428160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc427fff700 1 -- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc428160a10 msgr2=0x7fc428160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc427fff700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc428160a10 0x7fc428160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc427fff700 1 -- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc428154bd0 msgr2=0x7fc428153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc427fff700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc428154bd0 0x7fc428153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc427fff700 1 -- 172.21.15.89:0/2866785700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4281555f0 con 0x7fc410004850 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc4277fe700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc428154bd0 0x7fc428153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc42c817700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc428160a10 0x7fc428160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc427fff700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc428159aa0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0119a0 tx=0x7fc41c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc424ff9700 1 -- 172.21.15.89:0/2866785700 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c00f940 con 0x7fc410004850 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc424ff9700 1 -- 172.21.15.89:0/2866785700 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c00faa0 con 0x7fc410004850 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4281558d0 con 0x7fc410004850 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc424ff9700 1 -- 172.21.15.89:0/2866785700 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c00fe30 con 0x7fc410004850 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.186+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc428155e90 con 0x7fc410004850 2023-11-24T06:20:39.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.187+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4280753f0 con 0x7fc410004850 2023-11-24T06:20:39.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.187+0000 7fc424ff9700 1 -- 172.21.15.89:0/2866785700 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fc41c014440 con 0x7fc410004850 2023-11-24T06:20:39.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.188+0000 7fc424ff9700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc41406af60 0x7fc41406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.188+0000 7fc424ff9700 1 -- 172.21.15.89:0/2866785700 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc41c089b90 con 0x7fc410004850 2023-11-24T06:20:39.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.188+0000 7fc4277fe700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc41406af60 0x7fc41406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.188+0000 7fc4277fe700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc41406af60 0x7fc41406d420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc4281594e0 tx=0x7fc41800a000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:39.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.192+0000 7fc424ff9700 1 -- 172.21.15.89:0/2866785700 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc41c054d90 con 0x7fc410004850 2023-11-24T06:20:39.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.395+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc4280c5bd0 con 0x7fc410004850 2023-11-24T06:20:39.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.396+0000 7fc424ff9700 1 -- 172.21.15.89:0/2866785700 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v53) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc4280c5bd0 con 0x7fc410004850 2023-11-24T06:20:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.397+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc41406af60 msgr2=0x7fc41406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.397+0000 7fc42e27a700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc41406af60 0x7fc41406d420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc4281594e0 tx=0x7fc41800a000 comp rx=0 tx=0).stop 2023-11-24T06:20:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.397+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 msgr2=0x7fc428159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.397+0000 7fc42e27a700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc428159aa0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0119a0 tx=0x7fc41c011d60 comp rx=0 tx=0).stop 2023-11-24T06:20:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.398+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 shutdown_connections 2023-11-24T06:20:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.398+0000 7fc42e27a700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc428160a10 0x7fc428160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.398+0000 7fc42e27a700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc410004850 0x7fc428159aa0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.398+0000 7fc42e27a700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc41406af60 0x7fc41406d420 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.398+0000 7fc42e27a700 1 --2- 172.21.15.89:0/2866785700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc428154bd0 0x7fc428153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.398+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 >> 172.21.15.89:0/2866785700 conn(0x7fc428071b90 msgr2=0x7fc4280b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:39.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.398+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 shutdown_connections 2023-11-24T06:20:39.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.398+0000 7fc42e27a700 1 -- 172.21.15.89:0/2866785700 wait complete. 2023-11-24T06:20:39.402 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2023-11-24T06:20:39.412 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T06:20:39.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:39.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T06:20:39.454 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:20:39.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:39.455 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T06:20:39.511 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:39.512 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:39.533 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:39.533 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:39.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:39.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.712+0000 7fde97ba5700 1 Processor -- start 2023-11-24T06:20:39.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.712+0000 7fde97ba5700 1 -- start start 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.713+0000 7fde97ba5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fde900d0c40 0x7fde900ca190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.713+0000 7fde97ba5700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900d10d0 con 0x7fde900b0fa0 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.713+0000 7fde97ba5700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900622a0 con 0x7fde900afc80 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.713+0000 7fde97ba5700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde900ca6d0 con 0x7fde900d0c40 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.713+0000 7fde95941700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fde900b0fa0 0x7fde900b1380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41848/0 (socket says 172.21.15.89:41848) 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.713+0000 7fde95941700 1 -- 172.21.15.89:0/2289786351 learned_addr learned my addr 172.21.15.89:0/2289786351 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.713+0000 7fde96142700 1 --2- 172.21.15.89:0/2289786351 >> v2:172.21.15.89:3300/0 conn(0x7fde900d0c40 0x7fde900ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.713+0000 7fde9493f700 1 -- 172.21.15.89:0/2289786351 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046137258 0 0) 0x7fde900d10d0 con 0x7fde900b0fa0 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde9493f700 1 -- 172.21.15.89:0/2289786351 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde7c0035b0 con 0x7fde900b0fa0 2023-11-24T06:20:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde96142700 1 -- 172.21.15.89:0/2289786351 >> v1:172.21.15.123:6789/0 conn(0x7fde900afc80 legacy=0x7fde900ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde96142700 1 -- 172.21.15.89:0/2289786351 >> v1:172.21.15.89:6789/0 conn(0x7fde900b0fa0 legacy=0x7fde900b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde96142700 1 -- 172.21.15.89:0/2289786351 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde90062580 con 0x7fde900d0c40 2023-11-24T06:20:39.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde96142700 1 --2- 172.21.15.89:0/2289786351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde900ca190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fde88009d90 tx=0x7fde8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e9ec09616a7f0e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:39.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde9493f700 1 -- 172.21.15.89:0/2289786351 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8800e070 con 0x7fde900d0c40 2023-11-24T06:20:39.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde9493f700 1 -- 172.21.15.89:0/2289786351 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8800bb20 con 0x7fde900d0c40 2023-11-24T06:20:39.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde9493f700 1 -- 172.21.15.89:0/2289786351 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88012430 con 0x7fde900d0c40 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde97ba5700 1 -- 172.21.15.89:0/2289786351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 msgr2=0x7fde900ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde97ba5700 1 --2- 172.21.15.89:0/2289786351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde900ca190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fde88009d90 tx=0x7fde8800b330 comp rx=0 tx=0).stop 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde97ba5700 1 -- 172.21.15.89:0/2289786351 shutdown_connections 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde97ba5700 1 --2- 172.21.15.89:0/2289786351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde900ca190 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.714+0000 7fde97ba5700 1 -- 172.21.15.89:0/2289786351 >> 172.21.15.89:0/2289786351 conn(0x7fde900b57b0 msgr2=0x7fde900b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.715+0000 7fde97ba5700 1 -- 172.21.15.89:0/2289786351 shutdown_connections 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.715+0000 7fde97ba5700 1 -- 172.21.15.89:0/2289786351 wait complete. 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.716+0000 7fde97ba5700 1 Processor -- start 2023-11-24T06:20:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.716+0000 7fde97ba5700 1 -- start start 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.716+0000 7fde97ba5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde900afc80 0x7fde90150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.716+0000 7fde97ba5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde900b0fa0 0x7fde900cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.716+0000 7fde97ba5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde90157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.716+0000 7fde97ba5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde900cd0d0 con 0x7fde900d0c40 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.716+0000 7fde97ba5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde901508c0 con 0x7fde900b0fa0 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.716+0000 7fde97ba5700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde900cc4c0 con 0x7fde900afc80 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde90157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde90157190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38354/0 (socket says 172.21.15.89:38354) 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 -- 172.21.15.89:0/1343288517 learned_addr learned my addr 172.21.15.89:0/1343288517 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde95140700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde900b0fa0 0x7fde900cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde95941700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde900afc80 0x7fde90150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 -- 172.21.15.89:0/1343288517 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde900afc80 msgr2=0x7fde90150380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde900afc80 0x7fde90150380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 -- 172.21.15.89:0/1343288517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde900b0fa0 msgr2=0x7fde900cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde900b0fa0 0x7fde900cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 -- 172.21.15.89:0/1343288517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde88009a70 con 0x7fde900d0c40 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde96142700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde90157190 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fde88014040 tx=0x7fde88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde867fc700 1 -- 172.21.15.89:0/1343288517 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88012430 con 0x7fde900d0c40 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde900cc7a0 con 0x7fde900d0c40 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde867fc700 1 -- 172.21.15.89:0/1343288517 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde88020070 con 0x7fde900d0c40 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde867fc700 1 -- 172.21.15.89:0/1343288517 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88004280 con 0x7fde900d0c40 2023-11-24T06:20:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.717+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde900ccd60 con 0x7fde900d0c40 2023-11-24T06:20:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.718+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde900cd240 con 0x7fde900d0c40 2023-11-24T06:20:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.719+0000 7fde867fc700 1 -- 172.21.15.89:0/1343288517 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fde880043e0 con 0x7fde900d0c40 2023-11-24T06:20:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.719+0000 7fde867fc700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fde7406af60 0x7fde7406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.719+0000 7fde95941700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fde7406af60 0x7fde7406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.719+0000 7fde867fc700 1 -- 172.21.15.89:0/1343288517 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde88022070 con 0x7fde900d0c40 2023-11-24T06:20:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.719+0000 7fde95941700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fde7406af60 0x7fde7406d420 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fde80007fd0 tx=0x7fde80007dc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.724+0000 7fde867fc700 1 -- 172.21.15.89:0/1343288517 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde88056300 con 0x7fde900d0c40 2023-11-24T06:20:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.915+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde90067f60 con 0x7fde900d0c40 2023-11-24T06:20:39.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.916+0000 7fde867fc700 1 -- 172.21.15.89:0/1343288517 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16529 (secure 0 0 0) 0x7fde88059920 con 0x7fde900d0c40 2023-11-24T06:20:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:39.920 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":53,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:23.644194+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:20.413276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:20.978712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fde7406af60 msgr2=0x7fde7406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fde7406af60 0x7fde7406d420 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fde80007fd0 tx=0x7fde80007dc0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 msgr2=0x7fde90157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde90157190 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fde88014040 tx=0x7fde88004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 shutdown_connections 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde900d0c40 0x7fde90157190 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde900afc80 0x7fde90150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fde7406af60 0x7fde7406d420 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 --2- 172.21.15.89:0/1343288517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde900b0fa0 0x7fde900cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.918+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 >> 172.21.15.89:0/1343288517 conn(0x7fde900b57b0 msgr2=0x7fde900b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.919+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 shutdown_connections 2023-11-24T06:20:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:39.919+0000 7fde97ba5700 1 -- 172.21.15.89:0/1343288517 wait complete. 2023-11-24T06:20:39.934 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:20:20.413276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:20:20.978712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:39.934 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:40.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.057+0000 7f6b4b55a700 1 Processor -- start 2023-11-24T06:20:40.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.057+0000 7f6b4b55a700 1 -- start start 2023-11-24T06:20:40.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b4b55a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6b4415e160 0x7f6b440b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b4b55a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b44073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b4b55a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b440744d0 0x7f6b4406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b4b55a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b440b4500 con 0x7f6b440744d0 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b4b55a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b44074960 con 0x7f6b4415e160 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b4b55a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440b3c60 con 0x7f6b440afc90 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b440744d0 0x7f6b4406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b440744d0 0x7f6b4406d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38362/0 (socket says 172.21.15.89:38362) 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 -- 172.21.15.89:0/458342450 learned_addr learned my addr 172.21.15.89:0/458342450 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b492f6700 1 --2- 172.21.15.89:0/458342450 >> v2:172.21.15.123:3300/0 conn(0x7f6b4415e160 0x7f6b440b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b48af5700 1 --2- 172.21.15.89:0/458342450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b44073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 -- 172.21.15.89:0/458342450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 msgr2=0x7f6b44073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 --2- 172.21.15.89:0/458342450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b44073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 -- 172.21.15.89:0/458342450 >> v2:172.21.15.123:3300/0 conn(0x7f6b4415e160 msgr2=0x7f6b440b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 --2- 172.21.15.89:0/458342450 >> v2:172.21.15.123:3300/0 conn(0x7f6b4415e160 0x7f6b440b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 -- 172.21.15.89:0/458342450 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b440b58c0 con 0x7f6b440744d0 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b492f6700 1 --2- 172.21.15.89:0/458342450 >> v2:172.21.15.123:3300/0 conn(0x7f6b4415e160 0x7f6b440b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b48af5700 1 --2- 172.21.15.89:0/458342450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b44073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.058+0000 7f6b49af7700 1 --2- 172.21.15.89:0/458342450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b440744d0 0x7f6b4406d930 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6b4409efa0 tx=0x7f6b3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b057cc17e6d6164 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b3bfff700 1 -- 172.21.15.89:0/458342450 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00e070 con 0x7f6b440744d0 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b3bfff700 1 -- 172.21.15.89:0/458342450 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c00b9c0 con 0x7f6b440744d0 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b3bfff700 1 -- 172.21.15.89:0/458342450 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00bd70 con 0x7f6b440744d0 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 -- 172.21.15.89:0/458342450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b440744d0 msgr2=0x7f6b4406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/458342450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b440744d0 0x7f6b4406d930 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6b4409efa0 tx=0x7f6b3c00b330 comp rx=0 tx=0).stop 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 -- 172.21.15.89:0/458342450 shutdown_connections 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/458342450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b440744d0 0x7f6b4406d930 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/458342450 >> v2:172.21.15.123:3300/0 conn(0x7f6b4415e160 0x7f6b440b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/458342450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b44073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 -- 172.21.15.89:0/458342450 >> 172.21.15.89:0/458342450 conn(0x7f6b440d0bf0 msgr2=0x7f6b440d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 -- 172.21.15.89:0/458342450 shutdown_connections 2023-11-24T06:20:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.059+0000 7f6b4b55a700 1 -- 172.21.15.89:0/458342450 wait complete. 2023-11-24T06:20:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.060+0000 7f6b4b55a700 1 Processor -- start 2023-11-24T06:20:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.060+0000 7f6b4b55a700 1 -- start start 2023-11-24T06:20:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b4b55a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b440744d0 0x7f6b440b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b4b55a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b4406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b4b55a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b4415e160 0x7f6b4406de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b4b55a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440b1340 con 0x7f6b4415e160 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b4b55a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440b0730 con 0x7f6b440744d0 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b4b55a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440b0a10 con 0x7f6b440afc90 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b49af7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b4415e160 0x7f6b4406de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b49af7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b4415e160 0x7f6b4406de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38368/0 (socket says 172.21.15.89:38368) 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b49af7700 1 -- 172.21.15.89:0/732495808 learned_addr learned my addr 172.21.15.89:0/732495808 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b492f6700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b440744d0 0x7f6b440b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b48af5700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b4406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b49af7700 1 -- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 msgr2=0x7f6b4406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.061+0000 7f6b49af7700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b4406d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b49af7700 1 -- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b440744d0 msgr2=0x7f6b440b1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b49af7700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b440744d0 0x7f6b440b1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b49af7700 1 -- 172.21.15.89:0/732495808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c009b30 con 0x7f6b4415e160 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b492f6700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b440744d0 0x7f6b440b1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b49af7700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b4415e160 0x7f6b4406de70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6b44072ed0 tx=0x7f6b3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b39ffb700 1 -- 172.21.15.89:0/732495808 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c0092e0 con 0x7f6b4415e160 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b39ffb700 1 -- 172.21.15.89:0/732495808 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c01c440 con 0x7f6b4415e160 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b39ffb700 1 -- 172.21.15.89:0/732495808 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c01c7f0 con 0x7f6b4415e160 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b440b0cf0 con 0x7f6b4415e160 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.062+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b440b0350 con 0x7f6b4415e160 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.063+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4415e540 con 0x7f6b4415e160 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.063+0000 7f6b39ffb700 1 -- 172.21.15.89:0/732495808 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f6b3c01c950 con 0x7f6b4415e160 2023-11-24T06:20:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.064+0000 7f6b39ffb700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6b2806af60 0x7f6b2806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.064+0000 7f6b39ffb700 1 -- 172.21.15.89:0/732495808 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b3c08b0d0 con 0x7f6b4415e160 2023-11-24T06:20:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.064+0000 7f6b492f6700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6b2806af60 0x7f6b2806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.064+0000 7f6b492f6700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6b2806af60 0x7f6b2806d420 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6b340099b0 tx=0x7f6b34008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.068+0000 7f6b39ffb700 1 -- 172.21.15.89:0/732495808 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b3c056110 con 0x7f6b4415e160 2023-11-24T06:20:40.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.258+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b440b04e0 con 0x7f6b4415e160 2023-11-24T06:20:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.259+0000 7f6b39ffb700 1 -- 172.21.15.89:0/732495808 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b3c059730 con 0x7f6b4415e160 2023-11-24T06:20:40.262 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:40.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6b2806af60 msgr2=0x7f6b2806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6b2806af60 0x7f6b2806d420 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6b340099b0 tx=0x7f6b34008040 comp rx=0 tx=0).stop 2023-11-24T06:20:40.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b4415e160 msgr2=0x7f6b4406de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b4415e160 0x7f6b4406de70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6b44072ed0 tx=0x7f6b3c004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 shutdown_connections 2023-11-24T06:20:40.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b4415e160 0x7f6b4406de70 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6b2806af60 0x7f6b2806d420 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b440744d0 0x7f6b440b1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 --2- 172.21.15.89:0/732495808 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b440afc90 0x7f6b4406d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 >> 172.21.15.89:0/732495808 conn(0x7f6b440d0bf0 msgr2=0x7f6b4406ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:40.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 shutdown_connections 2023-11-24T06:20:40.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.261+0000 7f6b4b55a700 1 -- 172.21.15.89:0/732495808 wait complete. 2023-11-24T06:20:40.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:20:40.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.397+0000 7f3b49831700 1 Processor -- start 2023-11-24T06:20:40.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.397+0000 7f3b49831700 1 -- start start 2023-11-24T06:20:40.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b49831700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b4415dfa0 0x7f3b4406ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b49831700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4406bf40 con 0x7f3b440b6ec0 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b49831700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b440b7a30 con 0x7f3b440cff20 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b49831700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4406c210 con 0x7f3b4415dfa0 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b427fc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3b440b6ec0 0x7f3b440b7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41876/0 (socket says 172.21.15.89:41876) 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b427fc700 1 -- 172.21.15.89:0/2673940226 learned_addr learned my addr 172.21.15.89:0/2673940226 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b42ffd700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b4415dfa0 0x7f3b4406ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648043407 0 0) 0x7f3b4406bf40 con 0x7f3b440b6ec0 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2c0035b0 con 0x7f3b440b6ec0 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114096133 0 0) 0x7f3b440b7a30 con 0x7f3b440cff20 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b4406bf40 con 0x7f3b440cff20 2023-11-24T06:20:40.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2887823491 0 0) 0x7f3b2c0035b0 con 0x7f3b440b6ec0 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b4415dfa0 msgr2=0x7f3b4406ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b4415dfa0 0x7f3b4406ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 >> v1:172.21.15.123:6789/0 conn(0x7f3b440cff20 legacy=0x7f3b440d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b440b4b30 con 0x7f3b440b6ec0 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f3b38001670 con 0x7f3b440b6ec0 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 >> v1:172.21.15.89:6789/0 conn(0x7f3b440b6ec0 legacy=0x7f3b440b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b4415dfa0 0x7f3b2c003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b2c004120 0x7f3b2c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004d90 0x7f3b2c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4406c210 con 0x7f3b2c004120 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c00baf0 con 0x7f3b2c004d90 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c00bcb0 con 0x7f3b4415dfa0 2023-11-24T06:20:40.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b427fc700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b2c004120 0x7f3b2c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b42ffd700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b4415dfa0 0x7f3b2c003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.398+0000 7f3b437fe700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004d90 0x7f3b2c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b427fc700 1 -- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b4415dfa0 msgr2=0x7f3b2c003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b427fc700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b4415dfa0 0x7f3b2c003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b427fc700 1 -- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004d90 msgr2=0x7f3b2c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b427fc700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004d90 0x7f3b2c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b427fc700 1 -- 172.21.15.89:0/2673940226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0035b0 con 0x7f3b2c004120 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b437fe700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004d90 0x7f3b2c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b427fc700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b2c004120 0x7f3b2c004850 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c210 tx=0x7f3b2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48558e7eb0d316c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3800cd50 con 0x7f3b2c004120 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b38001af0 con 0x7f3b2c004120 2023-11-24T06:20:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b41ffb700 1 -- 172.21.15.89:0/2673940226 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38005ba0 con 0x7f3b2c004120 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b49831700 1 -- 172.21.15.89:0/2673940226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b2c004120 msgr2=0x7f3b2c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.399+0000 7f3b49831700 1 --2- 172.21.15.89:0/2673940226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b2c004120 0x7f3b2c004850 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c210 tx=0x7f3b2c005880 comp rx=0 tx=0).stop 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.400+0000 7f3b437fe700 1 -- 172.21.15.89:0/2673940226 reap_dead start 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.400+0000 7f3b49831700 1 -- 172.21.15.89:0/2673940226 shutdown_connections 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.400+0000 7f3b49831700 1 -- 172.21.15.89:0/2673940226 >> 172.21.15.89:0/2673940226 conn(0x7f3b440b3810 msgr2=0x7f3b440b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.400+0000 7f3b49831700 1 -- 172.21.15.89:0/2673940226 shutdown_connections 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.400+0000 7f3b49831700 1 -- 172.21.15.89:0/2673940226 wait complete. 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.401+0000 7f3b49831700 1 Processor -- start 2023-11-24T06:20:40.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.401+0000 7f3b49831700 1 -- start start 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b49831700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004120 0x7f3b4415a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b49831700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b4409dc70 0x7f3b4409e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b49831700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b440c7ac0 0x7f3b440c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b49831700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b440c8480 con 0x7f3b4409dc70 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b49831700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b440c65c0 con 0x7f3b2c004120 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b49831700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b440c68a0 con 0x7f3b440c7ac0 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b4409dc70 0x7f3b4409e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b4409dc70 0x7f3b4409e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38386/0 (socket says 172.21.15.89:38386) 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 -- 172.21.15.89:0/1471738018 learned_addr learned my addr 172.21.15.89:0/1471738018 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:40.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b42ffd700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004120 0x7f3b4415a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b437fe700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b440c7ac0 0x7f3b440c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 -- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b440c7ac0 msgr2=0x7f3b440c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b440c7ac0 0x7f3b440c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 -- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004120 msgr2=0x7f3b4415a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004120 0x7f3b4415a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 -- 172.21.15.89:0/1471738018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b440c6b80 con 0x7f3b4409dc70 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b42ffd700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004120 0x7f3b4415a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b427fc700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b4409dc70 0x7f3b4409e0f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00bac0 tx=0x7f3b38001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b2bfff700 1 -- 172.21.15.89:0/1471738018 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38004d00 con 0x7f3b4409dc70 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b2bfff700 1 -- 172.21.15.89:0/1471738018 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b38004e60 con 0x7f3b4409dc70 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b440c6e60 con 0x7f3b4409dc70 2023-11-24T06:20:40.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b2bfff700 1 -- 172.21.15.89:0/1471738018 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38007310 con 0x7f3b4409dc70 2023-11-24T06:20:40.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.402+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4415a740 con 0x7f3b4409dc70 2023-11-24T06:20:40.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.403+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b44070400 con 0x7f3b4409dc70 2023-11-24T06:20:40.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.404+0000 7f3b2bfff700 1 -- 172.21.15.89:0/1471738018 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f3b38007470 con 0x7f3b4409dc70 2023-11-24T06:20:40.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.404+0000 7f3b2bfff700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b3006af60 0x7f3b3006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.404+0000 7f3b42ffd700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b3006af60 0x7f3b3006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.404+0000 7f3b2bfff700 1 -- 172.21.15.89:0/1471738018 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b3808a2f0 con 0x7f3b4409dc70 2023-11-24T06:20:40.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.405+0000 7f3b42ffd700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b3006af60 0x7f3b3006d420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009b80 tx=0x7f3b34008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.409+0000 7f3b2bfff700 1 -- 172.21.15.89:0/1471738018 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b380553c0 con 0x7f3b4409dc70 2023-11-24T06:20:40.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.599+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3b440b7c60 con 0x7f3b4409dc70 2023-11-24T06:20:40.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.599+0000 7f3b2bfff700 1 -- 172.21.15.89:0/1471738018 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3b380589e0 con 0x7f3b4409dc70 2023-11-24T06:20:40.603 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:20:40.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b3006af60 msgr2=0x7f3b3006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b3006af60 0x7f3b3006d420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009b80 tx=0x7f3b34008040 comp rx=0 tx=0).stop 2023-11-24T06:20:40.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b4409dc70 msgr2=0x7f3b4409e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b4409dc70 0x7f3b4409e0f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00bac0 tx=0x7f3b38001980 comp rx=0 tx=0).stop 2023-11-24T06:20:40.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 shutdown_connections 2023-11-24T06:20:40.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3b440c7ac0 0x7f3b440c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3b3006af60 0x7f3b3006d420 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b2c004120 0x7f3b4415a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 --2- 172.21.15.89:0/1471738018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b4409dc70 0x7f3b4409e0f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 >> 172.21.15.89:0/1471738018 conn(0x7f3b440b3810 msgr2=0x7f3b4406fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:40.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 shutdown_connections 2023-11-24T06:20:40.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.602+0000 7f3b49831700 1 -- 172.21.15.89:0/1471738018 wait complete. 2023-11-24T06:20:40.618 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:20:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.738+0000 7f052d137700 1 Processor -- start 2023-11-24T06:20:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.738+0000 7f052d137700 1 -- start start 2023-11-24T06:20:40.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052d137700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f05280b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052d137700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05280b3c60 con 0x7f05280afc90 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052d137700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05280b58c0 con 0x7f052815e160 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052d137700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05280b4500 con 0x7f05280746a0 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052759e700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f05280afc90 0x7f052806d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41900/0 (socket says 172.21.15.89:41900) 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052759e700 1 -- 172.21.15.89:0/2341186590 learned_addr learned my addr 172.21.15.89:0/2341186590 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052659c700 1 --2- 172.21.15.89:0/2341186590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f05280b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f0525d9b700 1 -- 172.21.15.89:0/2341186590 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967848282 0 0) 0x7f05280b3c60 con 0x7f05280afc90 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f0525d9b700 1 -- 172.21.15.89:0/2341186590 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05100035b0 con 0x7f05280afc90 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f0525d9b700 1 -- 172.21.15.89:0/2341186590 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329766522 0 0) 0x7f05280b58c0 con 0x7f052815e160 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f0525d9b700 1 -- 172.21.15.89:0/2341186590 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05280b3c60 con 0x7f052815e160 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f0525d9b700 1 -- 172.21.15.89:0/2341186590 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353522670 0 0) 0x7f05100035b0 con 0x7f05280afc90 2023-11-24T06:20:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052659c700 1 -- 172.21.15.89:0/2341186590 >> v1:172.21.15.123:6789/0 conn(0x7f052815e160 legacy=0x7f0528073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052659c700 1 -- 172.21.15.89:0/2341186590 >> v1:172.21.15.89:6789/0 conn(0x7f05280afc90 legacy=0x7f052806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052659c700 1 -- 172.21.15.89:0/2341186590 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05280b0140 con 0x7f05280746a0 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.739+0000 7f052659c700 1 --2- 172.21.15.89:0/2341186590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f05280b5da0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f051c009870 tx=0x7f051c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cc23878530aad8b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f0525d9b700 1 -- 172.21.15.89:0/2341186590 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520001970 con 0x7f05280746a0 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f0525d9b700 1 -- 172.21.15.89:0/2341186590 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051c004030 con 0x7f05280746a0 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f0525d9b700 1 -- 172.21.15.89:0/2341186590 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c0043a0 con 0x7f05280746a0 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f052d137700 1 -- 172.21.15.89:0/2341186590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 msgr2=0x7f05280b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f052d137700 1 --2- 172.21.15.89:0/2341186590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f05280b5da0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f051c009870 tx=0x7f051c009b80 comp rx=0 tx=0).stop 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f052d137700 1 -- 172.21.15.89:0/2341186590 shutdown_connections 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f052d137700 1 --2- 172.21.15.89:0/2341186590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f05280b5da0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f052d137700 1 -- 172.21.15.89:0/2341186590 >> 172.21.15.89:0/2341186590 conn(0x7f05280d0bf0 msgr2=0x7f05280d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f052d137700 1 -- 172.21.15.89:0/2341186590 shutdown_connections 2023-11-24T06:20:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.740+0000 7f052d137700 1 -- 172.21.15.89:0/2341186590 wait complete. 2023-11-24T06:20:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052d137700 1 Processor -- start 2023-11-24T06:20:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052d137700 1 -- start start 2023-11-24T06:20:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052d137700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f0528158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052d137700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05280afc90 0x7f052806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052d137700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f052815e160 0x7f052806ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052d137700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05281f4b60 con 0x7f05280afc90 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052d137700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0528159070 con 0x7f052815e160 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052d137700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05281f3ee0 con 0x7f05280746a0 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05280afc90 0x7f052806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05280afc90 0x7f052806e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38400/0 (socket says 172.21.15.89:38400) 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.742+0000 7f052659c700 1 -- 172.21.15.89:0/786846395 learned_addr learned my addr 172.21.15.89:0/786846395 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f0526d9d700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f0528158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052759e700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f052815e160 0x7f052806ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052659c700 1 -- 172.21.15.89:0/786846395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 msgr2=0x7f0528158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052659c700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f0528158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052659c700 1 -- 172.21.15.89:0/786846395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f052815e160 msgr2=0x7f052806ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052659c700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f052815e160 0x7f052806ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052659c700 1 -- 172.21.15.89:0/786846395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c0094a0 con 0x7f05280afc90 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052659c700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05280afc90 0x7f052806e7c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c00 tx=0x7f051c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f050f7fe700 1 -- 172.21.15.89:0/786846395 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c004570 con 0x7f05280afc90 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f050f7fe700 1 -- 172.21.15.89:0/786846395 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051c01f070 con 0x7f05280afc90 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05281f41c0 con 0x7f05280afc90 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f050f7fe700 1 -- 172.21.15.89:0/786846395 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c011870 con 0x7f05280afc90 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.743+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05281f4780 con 0x7f05280afc90 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.744+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05280c5bd0 con 0x7f05280afc90 2023-11-24T06:20:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.744+0000 7f050f7fe700 1 -- 172.21.15.89:0/786846395 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f051c0119d0 con 0x7f05280afc90 2023-11-24T06:20:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.745+0000 7f050f7fe700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f051406af60 0x7f051406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.745+0000 7f050f7fe700 1 -- 172.21.15.89:0/786846395 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f051c021070 con 0x7f05280afc90 2023-11-24T06:20:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.745+0000 7f0526d9d700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f051406af60 0x7f051406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.745+0000 7f0526d9d700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f051406af60 0x7f051406d420 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f05280b2930 tx=0x7f0518009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:40.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.749+0000 7f050f7fe700 1 -- 172.21.15.89:0/786846395 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05280c5bd0 con 0x7f05280afc90 2023-11-24T06:20:40.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.943+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05280c5bd0 con 0x7f05280afc90 2023-11-24T06:20:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.943+0000 7f050f7fe700 1 -- 172.21.15.89:0/786846395 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7f05280c5bd0 con 0x7f05280afc90 2023-11-24T06:20:40.946 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:20:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f051406af60 msgr2=0x7f051406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f051406af60 0x7f051406d420 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f05280b2930 tx=0x7f0518009040 comp rx=0 tx=0).stop 2023-11-24T06:20:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05280afc90 msgr2=0x7f052806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05280afc90 0x7f052806e7c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c00 tx=0x7f051c004750 comp rx=0 tx=0).stop 2023-11-24T06:20:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 shutdown_connections 2023-11-24T06:20:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f051406af60 0x7f051406d420 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f052815e160 0x7f052806ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05280746a0 0x7f0528158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 --2- 172.21.15.89:0/786846395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05280afc90 0x7f052806e7c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 >> 172.21.15.89:0/786846395 conn(0x7f05280d0bf0 msgr2=0x7f0528073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 shutdown_connections 2023-11-24T06:20:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:40.945+0000 7f052d137700 1 -- 172.21.15.89:0/786846395 wait complete. 2023-11-24T06:20:40.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:20:41.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.080+0000 7efe1823b700 1 Processor -- start 2023-11-24T06:20:41.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.080+0000 7efe1823b700 1 -- start start 2023-11-24T06:20:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe1823b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe101446a0 0x7efe100b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe1823b700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe100adf00 0x7efe1006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe1823b700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe100af950 con 0x7efe1006c540 2023-11-24T06:20:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe1823b700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1006c9f0 con 0x7efe101446a0 2023-11-24T06:20:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe1823b700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe10144a80 con 0x7efe100adf00 2023-11-24T06:20:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe15fd7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe101446a0 0x7efe100b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe167d8700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efe1006c540 0x7efe1006da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41912/0 (socket says 172.21.15.89:41912) 2023-11-24T06:20:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe167d8700 1 -- 172.21.15.89:0/2077785962 learned_addr learned my addr 172.21.15.89:0/2077785962 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe15fd7700 1 --2- 172.21.15.89:0/2077785962 >> v2:172.21.15.89:3300/0 conn(0x7efe101446a0 0x7efe100b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38406/0 (socket says 172.21.15.89:38406) 2023-11-24T06:20:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe157d6700 1 --2- 172.21.15.89:0/2077785962 >> v2:172.21.15.123:3300/0 conn(0x7efe100adf00 0x7efe1006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe15fd7700 1 -- 172.21.15.89:0/2077785962 >> v2:172.21.15.123:3300/0 conn(0x7efe100adf00 msgr2=0x7efe1006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:41.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe15fd7700 1 --2- 172.21.15.89:0/2077785962 >> v2:172.21.15.123:3300/0 conn(0x7efe100adf00 0x7efe1006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe15fd7700 1 -- 172.21.15.89:0/2077785962 >> v1:172.21.15.89:6789/0 conn(0x7efe1006c540 legacy=0x7efe1006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe15fd7700 1 -- 172.21.15.89:0/2077785962 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe100afc30 con 0x7efe101446a0 2023-11-24T06:20:41.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe15fd7700 1 --2- 172.21.15.89:0/2077785962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe101446a0 0x7efe100b5ae0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efe1009f030 tx=0x7efe0c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fa12009d81c890 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe14fd5700 1 -- 172.21.15.89:0/2077785962 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c011830 con 0x7efe101446a0 2023-11-24T06:20:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe14fd5700 1 -- 172.21.15.89:0/2077785962 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0c011990 con 0x7efe101446a0 2023-11-24T06:20:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.081+0000 7efe14fd5700 1 -- 172.21.15.89:0/2077785962 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c011d20 con 0x7efe101446a0 2023-11-24T06:20:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.082+0000 7efe1823b700 1 -- 172.21.15.89:0/2077785962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe101446a0 msgr2=0x7efe100b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.082+0000 7efe1823b700 1 --2- 172.21.15.89:0/2077785962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe101446a0 0x7efe100b5ae0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efe1009f030 tx=0x7efe0c009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.082+0000 7efe1823b700 1 -- 172.21.15.89:0/2077785962 shutdown_connections 2023-11-24T06:20:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.082+0000 7efe1823b700 1 --2- 172.21.15.89:0/2077785962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe101446a0 0x7efe100b5ae0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.082+0000 7efe1823b700 1 --2- 172.21.15.89:0/2077785962 >> v2:172.21.15.123:3300/0 conn(0x7efe100adf00 0x7efe1006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.082+0000 7efe1823b700 1 -- 172.21.15.89:0/2077785962 >> 172.21.15.89:0/2077785962 conn(0x7efe10071b90 msgr2=0x7efe10071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.082+0000 7efe1823b700 1 -- 172.21.15.89:0/2077785962 shutdown_connections 2023-11-24T06:20:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.082+0000 7efe1823b700 1 -- 172.21.15.89:0/2077785962 wait complete. 2023-11-24T06:20:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.083+0000 7efe1823b700 1 Processor -- start 2023-11-24T06:20:41.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.083+0000 7efe1823b700 1 -- start start 2023-11-24T06:20:41.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe1823b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe1006c540 0x7efe10143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe1823b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe100adf00 0x7efe1015a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe1823b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe101446a0 0x7efe101f7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe1823b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1015ac40 con 0x7efe100adf00 2023-11-24T06:20:41.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe1823b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe101f4a50 con 0x7efe101446a0 2023-11-24T06:20:41.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe1823b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe101f7fa0 con 0x7efe1006c540 2023-11-24T06:20:41.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe100adf00 0x7efe1015a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe100adf00 0x7efe1015a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38412/0 (socket says 172.21.15.89:38412) 2023-11-24T06:20:41.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 -- 172.21.15.89:0/4252210499 learned_addr learned my addr 172.21.15.89:0/4252210499 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:41.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe167d8700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe101446a0 0x7efe101f7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe15fd7700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe1006c540 0x7efe10143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 -- 172.21.15.89:0/4252210499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe1006c540 msgr2=0x7efe10143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe1006c540 0x7efe10143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 -- 172.21.15.89:0/4252210499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe101446a0 msgr2=0x7efe101f7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe101446a0 0x7efe101f7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 -- 172.21.15.89:0/4252210499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c0094a0 con 0x7efe100adf00 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.084+0000 7efe157d6700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe100adf00 0x7efe1015a700 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7efe0400e7e0 tx=0x7efe0400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.085+0000 7efe027fc700 1 -- 172.21.15.89:0/4252210499 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04013610 con 0x7efe100adf00 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.085+0000 7efe027fc700 1 -- 172.21.15.89:0/4252210499 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe04013770 con 0x7efe100adf00 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.085+0000 7efe027fc700 1 -- 172.21.15.89:0/4252210499 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04013b20 con 0x7efe100adf00 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.085+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe101f4490 con 0x7efe100adf00 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.085+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe101f7260 con 0x7efe100adf00 2023-11-24T06:20:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.086+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe100753f0 con 0x7efe100adf00 2023-11-24T06:20:41.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.086+0000 7efe027fc700 1 -- 172.21.15.89:0/4252210499 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7efe04013c80 con 0x7efe100adf00 2023-11-24T06:20:41.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.086+0000 7efe027fc700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efdf406af60 0x7efdf406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.086+0000 7efe027fc700 1 -- 172.21.15.89:0/4252210499 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe04089310 con 0x7efe100adf00 2023-11-24T06:20:41.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.086+0000 7efe15fd7700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efdf406af60 0x7efdf406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.087+0000 7efe15fd7700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efdf406af60 0x7efdf406d420 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0090b0 tx=0x7efe0c01c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.091+0000 7efe027fc700 1 -- 172.21.15.89:0/4252210499 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe04054350 con 0x7efe100adf00 2023-11-24T06:20:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.297+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efe10067f60 con 0x7efe100adf00 2023-11-24T06:20:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.297+0000 7efe027fc700 1 -- 172.21.15.89:0/4252210499 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v53) v1 ==== 80+0+0 (secure 0 0 0) 0x7efe04057970 con 0x7efe100adf00 2023-11-24T06:20:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efdf406af60 msgr2=0x7efdf406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efdf406af60 0x7efdf406d420 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0090b0 tx=0x7efe0c01c040 comp rx=0 tx=0).stop 2023-11-24T06:20:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe100adf00 msgr2=0x7efe1015a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe100adf00 0x7efe1015a700 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7efe0400e7e0 tx=0x7efe0400eba0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 shutdown_connections 2023-11-24T06:20:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efdf406af60 0x7efdf406d420 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe101446a0 0x7efe101f7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe1006c540 0x7efe10143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 --2- 172.21.15.89:0/4252210499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe100adf00 0x7efe1015a700 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 >> 172.21.15.89:0/4252210499 conn(0x7efe10071b90 msgr2=0x7efe1006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 shutdown_connections 2023-11-24T06:20:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.299+0000 7efe1823b700 1 -- 172.21.15.89:0/4252210499 wait complete. 2023-11-24T06:20:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2023-11-24T06:20:41.314 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T06:20:41.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:41.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T06:20:41.362 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:20:41.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:20:41.362 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T06:20:41.379 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:41.380 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:41.397 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:20:41.397 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:41.397 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:41.437 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:20:41.437 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.556+0000 7f8f0ffe1700 1 Processor -- start 2023-11-24T06:20:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.557+0000 7f8f0ffe1700 1 -- start start 2023-11-24T06:20:41.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.557+0000 7f8f0ffe1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f080adf00 0x7f8f0806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.557+0000 7f8f0ffe1700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8f0806c540 0x7f8f080b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.557+0000 7f8f0ffe1700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f080af950 con 0x7f8f081446a0 2023-11-24T06:20:41.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.557+0000 7f8f0ffe1700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f08144a80 con 0x7f8f0806c540 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.557+0000 7f8f0ffe1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0806c9b0 con 0x7f8f080adf00 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0d57c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f080adf00 0x7f8f0806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0e57e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8f0806c540 0x7f8f080b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0dd7d700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f8f081446a0 0x7f8f080b5ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:45744/0 (socket says 172.21.15.89:45744) 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0dd7d700 1 -- 172.21.15.89:0/1482634803 learned_addr learned my addr 172.21.15.89:0/1482634803 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0cd7b700 1 -- 172.21.15.89:0/1482634803 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009780454 0 0) 0x7f8f080af950 con 0x7f8f081446a0 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0cd7b700 1 -- 172.21.15.89:0/1482634803 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef40035b0 con 0x7f8f081446a0 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0d57c700 1 -- 172.21.15.89:0/1482634803 >> v2:172.21.15.123:3300/0 conn(0x7f8f0806c540 msgr2=0x7f8f080b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0d57c700 1 --2- 172.21.15.89:0/1482634803 >> v2:172.21.15.123:3300/0 conn(0x7f8f0806c540 0x7f8f080b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0d57c700 1 -- 172.21.15.89:0/1482634803 >> v1:172.21.15.123:6789/0 conn(0x7f8f081446a0 legacy=0x7f8f080b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0d57c700 1 -- 172.21.15.89:0/1482634803 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f080afc30 con 0x7f8f080adf00 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0e57e700 1 --2- 172.21.15.89:0/1482634803 >> v2:172.21.15.123:3300/0 conn(0x7f8f0806c540 0x7f8f080b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:41.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.558+0000 7f8f0d57c700 1 --2- 172.21.15.89:0/1482634803 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f080adf00 0x7f8f0806c000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009840 tx=0x7f8f04009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5229137a44d8abc9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0cd7b700 1 -- 172.21.15.89:0/1482634803 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04011830 con 0x7f8f080adf00 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0cd7b700 1 -- 172.21.15.89:0/1482634803 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f04011990 con 0x7f8f080adf00 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0cd7b700 1 -- 172.21.15.89:0/1482634803 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04011d00 con 0x7f8f080adf00 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/1482634803 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f080adf00 msgr2=0x7f8f0806c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/1482634803 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f080adf00 0x7f8f0806c000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009840 tx=0x7f8f04009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/1482634803 shutdown_connections 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/1482634803 >> v2:172.21.15.123:3300/0 conn(0x7f8f0806c540 0x7f8f080b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/1482634803 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f080adf00 0x7f8f0806c000 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/1482634803 >> 172.21.15.89:0/1482634803 conn(0x7f8f08071b90 msgr2=0x7f8f08071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/1482634803 shutdown_connections 2023-11-24T06:20:41.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.559+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/1482634803 wait complete. 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.560+0000 7f8f0ffe1700 1 Processor -- start 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.560+0000 7f8f0ffe1700 1 -- start start 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0ffe1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0806c540 0x7f8f080cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0ffe1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f080adf00 0x7f8f0815c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0ffe1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f081446a0 0x7f8f08153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0ffe1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080cd3b0 con 0x7f8f0806c540 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0ffe1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0815cb50 con 0x7f8f080adf00 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0ffe1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0815ce30 con 0x7f8f081446a0 2023-11-24T06:20:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0dd7d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0806c540 0x7f8f080cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0dd7d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0806c540 0x7f8f080cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38424/0 (socket says 172.21.15.89:38424) 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0dd7d700 1 -- 172.21.15.89:0/516164892 learned_addr learned my addr 172.21.15.89:0/516164892 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0d57c700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f080adf00 0x7f8f0815c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.561+0000 7f8f0e57e700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f081446a0 0x7f8f08153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0dd7d700 1 -- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f081446a0 msgr2=0x7f8f08153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0dd7d700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f081446a0 0x7f8f08153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0dd7d700 1 -- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f080adf00 msgr2=0x7f8f0815c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0dd7d700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f080adf00 0x7f8f0815c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0dd7d700 1 -- 172.21.15.89:0/516164892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f00009a70 con 0x7f8f0806c540 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0d57c700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f080adf00 0x7f8f0815c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0dd7d700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0806c540 0x7f8f080cc190 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8001eb0 tx=0x7f8ef800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8efe7fc700 1 -- 172.21.15.89:0/516164892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef800cea0 con 0x7f8f0806c540 2023-11-24T06:20:41.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8efe7fc700 1 -- 172.21.15.89:0/516164892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef8005d70 con 0x7f8f0806c540 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8efe7fc700 1 -- 172.21.15.89:0/516164892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80072f0 con 0x7f8f0806c540 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f040094a0 con 0x7f8f0806c540 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.562+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0815d3f0 con 0x7f8f0806c540 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.563+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f080753f0 con 0x7f8f0806c540 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.563+0000 7f8efe7fc700 1 -- 172.21.15.89:0/516164892 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f8ef8007450 con 0x7f8f0806c540 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.564+0000 7f8efe7fc700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eec06af60 0x7f8eec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.564+0000 7f8f0d57c700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eec06af60 0x7f8eec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.564+0000 7f8efe7fc700 1 -- 172.21.15.89:0/516164892 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ef80889f0 con 0x7f8f0806c540 2023-11-24T06:20:41.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.564+0000 7f8f0d57c700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eec06af60 0x7f8eec06d420 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8f04000c00 tx=0x7f8f04008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.568+0000 7f8efe7fc700 1 -- 172.21.15.89:0/516164892 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ef808ae60 con 0x7f8f0806c540 2023-11-24T06:20:41.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.800+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f08067f60 con 0x7f8f0806c540 2023-11-24T06:20:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.801+0000 7f8efe7fc700 1 -- 172.21.15.89:0/516164892 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f8ef8017090 con 0x7f8f0806c540 2023-11-24T06:20:41.804 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:41.804 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2011089843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2011089843},{"type":"v1","addr":"172.21.15.89:6836","nonce":2011089843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4955,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/2182962276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2182962276},{"type":"v1","addr":"172.21.15.123:6834","nonce":2182962276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2982219407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2982219407},{"type":"v1","addr":"172.21.15.123:6835","nonce":2982219407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:22.026873+0000","modified":"2023-11-24T06:20:25.053779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5577":{"gid":5577,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1341821445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1341821445},{"type":"v1","addr":"172.21.15.89:6837","nonce":1341821445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T06:20:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eec06af60 msgr2=0x7f8eec06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eec06af60 0x7f8eec06d420 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8f04000c00 tx=0x7f8f04008030 comp rx=0 tx=0).stop 2023-11-24T06:20:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0806c540 msgr2=0x7f8f080cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0806c540 0x7f8f080cc190 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8001eb0 tx=0x7f8ef800c7b0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 shutdown_connections 2023-11-24T06:20:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f081446a0 0x7f8f08153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0806c540 0x7f8f080cc190 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eec06af60 0x7f8eec06d420 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 --2- 172.21.15.89:0/516164892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f080adf00 0x7f8f0815c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 >> 172.21.15.89:0/516164892 conn(0x7f8f08071b90 msgr2=0x7f8f080b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 shutdown_connections 2023-11-24T06:20:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.803+0000 7f8f0ffe1700 1 -- 172.21.15.89:0/516164892 wait complete. 2023-11-24T06:20:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-11-24T06:20:41.818 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:41.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.935+0000 7ff068bdc700 1 Processor -- start 2023-11-24T06:20:41.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.935+0000 7ff068bdc700 1 -- start start 2023-11-24T06:20:41.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff068bdc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff06415e080 0x7ff0640b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff068bdc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 0x7ff0640afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff068bdc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0640ae430 con 0x7ff0640af4d0 2023-11-24T06:20:41.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff068bdc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0640af0b0 con 0x7ff06415e080 2023-11-24T06:20:41.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff068bdc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640b05a0 con 0x7ff06406c710 2023-11-24T06:20:41.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06259c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff06415e080 0x7ff0640b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff061d9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff0640af4d0 0x7ff06406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41938/0 (socket says 172.21.15.89:41938) 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff061d9b700 1 -- 172.21.15.89:0/779235464 learned_addr learned my addr 172.21.15.89:0/779235464 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff062d9d700 1 --2- 172.21.15.89:0/779235464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 0x7ff0640afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06259c700 1 -- 172.21.15.89:0/779235464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 msgr2=0x7ff0640afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06259c700 1 --2- 172.21.15.89:0/779235464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 0x7ff0640afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06259c700 1 -- 172.21.15.89:0/779235464 >> v1:172.21.15.89:6789/0 conn(0x7ff0640af4d0 legacy=0x7ff06406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06259c700 1 -- 172.21.15.89:0/779235464 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640ae710 con 0x7ff06415e080 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06159a700 1 -- 172.21.15.89:0/779235464 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128739744 0 0) 0x7ff0640ae430 con 0x7ff0640af4d0 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06259c700 1 --2- 172.21.15.89:0/779235464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff0640b5050 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff054009870 tx=0x7ff054009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40d69d1690a9788 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06159a700 1 -- 172.21.15.89:0/779235464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054011830 con 0x7ff06415e080 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06159a700 1 -- 172.21.15.89:0/779235464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff054011990 con 0x7ff06415e080 2023-11-24T06:20:41.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.936+0000 7ff06159a700 1 -- 172.21.15.89:0/779235464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054011d20 con 0x7ff06415e080 2023-11-24T06:20:41.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.937+0000 7ff068bdc700 1 -- 172.21.15.89:0/779235464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 msgr2=0x7ff0640b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.937+0000 7ff068bdc700 1 --2- 172.21.15.89:0/779235464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff0640b5050 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff054009870 tx=0x7ff054009b80 comp rx=0 tx=0).stop 2023-11-24T06:20:41.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.937+0000 7ff068bdc700 1 -- 172.21.15.89:0/779235464 shutdown_connections 2023-11-24T06:20:41.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.937+0000 7ff068bdc700 1 --2- 172.21.15.89:0/779235464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 0x7ff0640afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.937+0000 7ff068bdc700 1 --2- 172.21.15.89:0/779235464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff0640b5050 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.937+0000 7ff068bdc700 1 -- 172.21.15.89:0/779235464 >> 172.21.15.89:0/779235464 conn(0x7ff064071b90 msgr2=0x7ff064071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:41.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.937+0000 7ff068bdc700 1 -- 172.21.15.89:0/779235464 shutdown_connections 2023-11-24T06:20:41.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.937+0000 7ff068bdc700 1 -- 172.21.15.89:0/779235464 wait complete. 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.938+0000 7ff068bdc700 1 Processor -- start 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.938+0000 7ff068bdc700 1 -- start start 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff068bdc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 0x7ff0641f7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff068bdc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0640af4d0 0x7ff064159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff068bdc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff064159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff068bdc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff064158590 con 0x7ff06415e080 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff068bdc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0641f7d50 con 0x7ff0640af4d0 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff068bdc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff064158a20 con 0x7ff06406c710 2023-11-24T06:20:41.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff062d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff064159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff062d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff064159750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38442/0 (socket says 172.21.15.89:38442) 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.939+0000 7ff062d9d700 1 -- 172.21.15.89:0/709325823 learned_addr learned my addr 172.21.15.89:0/709325823 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff06259c700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 0x7ff0641f7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff061d9b700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0640af4d0 0x7ff064159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff062d9d700 1 -- 172.21.15.89:0/709325823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 msgr2=0x7ff0641f7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff062d9d700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 0x7ff0641f7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff062d9d700 1 -- 172.21.15.89:0/709325823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0640af4d0 msgr2=0x7ff064159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff062d9d700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0640af4d0 0x7ff064159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff062d9d700 1 -- 172.21.15.89:0/709325823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0540094a0 con 0x7ff06415e080 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff062d9d700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff064159750 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff05c007e30 tx=0x7ff05c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff052ffd700 1 -- 172.21.15.89:0/709325823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c00eea0 con 0x7ff06415e080 2023-11-24T06:20:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff052ffd700 1 -- 172.21.15.89:0/709325823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c014420 con 0x7ff06415e080 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff064157980 con 0x7ff06415e080 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff052ffd700 1 -- 172.21.15.89:0/709325823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0147d0 con 0x7ff06415e080 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.940+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff064157f40 con 0x7ff06415e080 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.941+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0640753f0 con 0x7ff06415e080 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.941+0000 7ff052ffd700 1 -- 172.21.15.89:0/709325823 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ff05c014930 con 0x7ff06415e080 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.942+0000 7ff052ffd700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff04406af60 0x7ff04406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.942+0000 7ff052ffd700 1 -- 172.21.15.89:0/709325823 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff05c089db0 con 0x7ff06415e080 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.942+0000 7ff06259c700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff04406af60 0x7ff04406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.942+0000 7ff06259c700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff04406af60 0x7ff04406d420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0540090b0 tx=0x7ff054008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:41.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:41.946+0000 7ff052ffd700 1 -- 172.21.15.89:0/709325823 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff05c054e70 con 0x7ff06415e080 2023-11-24T06:20:42.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.139+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff064067f60 con 0x7ff06415e080 2023-11-24T06:20:42.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.140+0000 7ff052ffd700 1 -- 172.21.15.89:0/709325823 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16529 (secure 0 0 0) 0x7ff05c058490 con 0x7ff06415e080 2023-11-24T06:20:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:42.144 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":53,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:23.644194+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:20.413276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:20.978712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:42.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff04406af60 msgr2=0x7ff04406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff04406af60 0x7ff04406d420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0540090b0 tx=0x7ff054008030 comp rx=0 tx=0).stop 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 msgr2=0x7ff064159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff064159750 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff05c007e30 tx=0x7ff05c00e3c0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 shutdown_connections 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff06415e080 0x7ff064159750 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff06406c710 0x7ff0641f7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff04406af60 0x7ff04406d420 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 --2- 172.21.15.89:0/709325823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0640af4d0 0x7ff064159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.142+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 >> 172.21.15.89:0/709325823 conn(0x7ff064071b90 msgr2=0x7ff0640b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.143+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 shutdown_connections 2023-11-24T06:20:42.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.143+0000 7ff068bdc700 1 -- 172.21.15.89:0/709325823 wait complete. 2023-11-24T06:20:42.158 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:20:20.413276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:20:20.978712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:42.159 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:42.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.279+0000 7f61ff16a700 1 Processor -- start 2023-11-24T06:20:42.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.279+0000 7f61ff16a700 1 -- start start 2023-11-24T06:20:42.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.280+0000 7f61ff16a700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f61f815e160 0x7f61f80b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.280+0000 7f61ff16a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f61f80afc90 0x7f61f8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.280+0000 7f61ff16a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80b3c60 con 0x7f61f80744d0 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.280+0000 7f61ff16a700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f80b4500 con 0x7f61f815e160 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.280+0000 7f61ff16a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f8074940 con 0x7f61f80afc90 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.280+0000 7f61fcf06700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f61f815e160 0x7f61f80b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.280+0000 7f61fd707700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f61f80744d0 0x7f61f806d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41952/0 (socket says 172.21.15.89:41952) 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.280+0000 7f61fd707700 1 -- 172.21.15.89:0/3223376771 learned_addr learned my addr 172.21.15.89:0/3223376771 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61f77fe700 1 -- 172.21.15.89:0/3223376771 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604840089 0 0) 0x7f61f80b3c60 con 0x7f61f80744d0 2023-11-24T06:20:42.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61f77fe700 1 -- 172.21.15.89:0/3223376771 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e80035b0 con 0x7f61f80744d0 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61f7fff700 1 --2- 172.21.15.89:0/3223376771 >> v2:172.21.15.123:3300/0 conn(0x7f61f80afc90 0x7f61f8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61fcf06700 1 -- 172.21.15.89:0/3223376771 >> v2:172.21.15.123:3300/0 conn(0x7f61f80afc90 msgr2=0x7f61f8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61fcf06700 1 --2- 172.21.15.89:0/3223376771 >> v2:172.21.15.123:3300/0 conn(0x7f61f80afc90 0x7f61f8073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61fcf06700 1 -- 172.21.15.89:0/3223376771 >> v1:172.21.15.89:6789/0 conn(0x7f61f80744d0 legacy=0x7f61f806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61fcf06700 1 -- 172.21.15.89:0/3223376771 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f80b58c0 con 0x7f61f815e160 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61f7fff700 1 --2- 172.21.15.89:0/3223376771 >> v2:172.21.15.123:3300/0 conn(0x7f61f80afc90 0x7f61f8073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61fcf06700 1 --2- 172.21.15.89:0/3223376771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f815e160 0x7f61f80b5da0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f61f80d0880 tx=0x7f61ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bba1e796165abc6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61f77fe700 1 -- 172.21.15.89:0/3223376771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec011830 con 0x7f61f815e160 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61f77fe700 1 -- 172.21.15.89:0/3223376771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec011990 con 0x7f61f815e160 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61f77fe700 1 -- 172.21.15.89:0/3223376771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec011d20 con 0x7f61f815e160 2023-11-24T06:20:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61ff16a700 1 -- 172.21.15.89:0/3223376771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f815e160 msgr2=0x7f61f80b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.281+0000 7f61ff16a700 1 --2- 172.21.15.89:0/3223376771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f815e160 0x7f61f80b5da0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f61f80d0880 tx=0x7f61ec009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:42.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.282+0000 7f61ff16a700 1 -- 172.21.15.89:0/3223376771 shutdown_connections 2023-11-24T06:20:42.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.282+0000 7f61ff16a700 1 --2- 172.21.15.89:0/3223376771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f815e160 0x7f61f80b5da0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.282+0000 7f61ff16a700 1 --2- 172.21.15.89:0/3223376771 >> v2:172.21.15.123:3300/0 conn(0x7f61f80afc90 0x7f61f8073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.282+0000 7f61ff16a700 1 -- 172.21.15.89:0/3223376771 >> 172.21.15.89:0/3223376771 conn(0x7f61f80d0bf0 msgr2=0x7f61f80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:42.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.282+0000 7f61ff16a700 1 -- 172.21.15.89:0/3223376771 shutdown_connections 2023-11-24T06:20:42.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.282+0000 7f61ff16a700 1 -- 172.21.15.89:0/3223376771 wait complete. 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.283+0000 7f61ff16a700 1 Processor -- start 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.283+0000 7f61ff16a700 1 -- start start 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61ff16a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80744d0 0x7f61f814ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61ff16a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f80afc90 0x7f61f8149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61ff16a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f815e160 0x7f61f80c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61ff16a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f8161d20 con 0x7f61f80744d0 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61ff16a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f814a390 con 0x7f61f80afc90 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61ff16a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f814d340 con 0x7f61f815e160 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80744d0 0x7f61f814ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80744d0 0x7f61f814ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38452/0 (socket says 172.21.15.89:38452) 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 -- 172.21.15.89:0/2354136416 learned_addr learned my addr 172.21.15.89:0/2354136416 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 -- 172.21.15.89:0/2354136416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f815e160 msgr2=0x7f61f80c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f815e160 0x7f61f80c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 -- 172.21.15.89:0/2354136416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f80afc90 msgr2=0x7f61f8149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61f7fff700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f80afc90 0x7f61f8149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f80afc90 0x7f61f8149ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 -- 172.21.15.89:0/2354136416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec0094a0 con 0x7f61f80744d0 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61fcf06700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80744d0 0x7f61f814ce00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0097c0 tx=0x7f61ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.284+0000 7f61f57fa700 1 -- 172.21.15.89:0/2354136416 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec0038a0 con 0x7f61f80744d0 2023-11-24T06:20:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.285+0000 7f61f57fa700 1 -- 172.21.15.89:0/2354136416 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec01e070 con 0x7f61f80744d0 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.285+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f81f7430 con 0x7f61f80744d0 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.285+0000 7f61f57fa700 1 -- 172.21.15.89:0/2354136416 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec003c40 con 0x7f61f80744d0 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.285+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f81f79f0 con 0x7f61f80744d0 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.286+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f8068fb0 con 0x7f61f80744d0 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.286+0000 7f61f57fa700 1 -- 172.21.15.89:0/2354136416 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f61ec02d450 con 0x7f61f80744d0 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.286+0000 7f61f57fa700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f61dc06af60 0x7f61dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.286+0000 7f61f57fa700 1 -- 172.21.15.89:0/2354136416 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61ec08a520 con 0x7f61f80744d0 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.286+0000 7f61f7fff700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f61dc06af60 0x7f61dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.287+0000 7f61f7fff700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f61dc06af60 0x7f61dc06d420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f61e4006fd0 tx=0x7f61e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.291+0000 7f61f57fa700 1 -- 172.21.15.89:0/2354136416 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61ec055670 con 0x7f61f80744d0 2023-11-24T06:20:42.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.480+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61f806c140 con 0x7f61f80744d0 2023-11-24T06:20:42.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.480+0000 7f61f57fa700 1 -- 172.21.15.89:0/2354136416 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61ec058c90 con 0x7f61f80744d0 2023-11-24T06:20:42.483 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:42.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f61dc06af60 msgr2=0x7f61dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f61dc06af60 0x7f61dc06d420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f61e4006fd0 tx=0x7f61e4008040 comp rx=0 tx=0).stop 2023-11-24T06:20:42.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80744d0 msgr2=0x7f61f814ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80744d0 0x7f61f814ce00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0097c0 tx=0x7f61ec004750 comp rx=0 tx=0).stop 2023-11-24T06:20:42.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 shutdown_connections 2023-11-24T06:20:42.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f815e160 0x7f61f80c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80744d0 0x7f61f814ce00 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f61dc06af60 0x7f61dc06d420 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 --2- 172.21.15.89:0/2354136416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f80afc90 0x7f61f8149ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 >> 172.21.15.89:0/2354136416 conn(0x7f61f80d0bf0 msgr2=0x7f61f815bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:42.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 shutdown_connections 2023-11-24T06:20:42.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.483+0000 7f61ff16a700 1 -- 172.21.15.89:0/2354136416 wait complete. 2023-11-24T06:20:42.498 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:20:42.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.619+0000 7f0bd2dec700 1 Processor -- start 2023-11-24T06:20:42.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.619+0000 7f0bd2dec700 1 -- start start 2023-11-24T06:20:42.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd2dec700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0bcc1587f0 0x7f0bcc06aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd2dec700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bcc0b4370 0x7f0bcc15b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd2dec700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc06b530 con 0x7f0bcc15b370 2023-11-24T06:20:42.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd2dec700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc0ae5e0 con 0x7f0bcc0b4370 2023-11-24T06:20:42.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd2dec700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bcc15be30 con 0x7f0bcc1587f0 2023-11-24T06:20:42.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd1389700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bcc0b4370 0x7f0bcc15b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd1389700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bcc0b4370 0x7f0bcc15b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38464/0 (socket says 172.21.15.89:38464) 2023-11-24T06:20:42.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd1389700 1 -- 172.21.15.89:0/237670358 learned_addr learned my addr 172.21.15.89:0/237670358 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:42.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd0b88700 1 --2- 172.21.15.89:0/237670358 >> v2:172.21.15.123:3300/0 conn(0x7f0bcc1587f0 0x7f0bcc06aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd1389700 1 -- 172.21.15.89:0/237670358 >> v2:172.21.15.123:3300/0 conn(0x7f0bcc1587f0 msgr2=0x7f0bcc06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:42.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd1389700 1 --2- 172.21.15.89:0/237670358 >> v2:172.21.15.123:3300/0 conn(0x7f0bcc1587f0 0x7f0bcc06aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd1389700 1 -- 172.21.15.89:0/237670358 >> v1:172.21.15.123:6789/0 conn(0x7f0bcc15b370 legacy=0x7f0bcc0b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:42.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd1389700 1 -- 172.21.15.89:0/237670358 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc15c110 con 0x7f0bcc0b4370 2023-11-24T06:20:42.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.620+0000 7f0bd1389700 1 --2- 172.21.15.89:0/237670358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc0b4370 0x7f0bcc15b800 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d150 tx=0x7f0bc400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7351c384acb11055 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bcb7fe700 1 -- 172.21.15.89:0/237670358 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4010070 con 0x7f0bcc0b4370 2023-11-24T06:20:42.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bcb7fe700 1 -- 172.21.15.89:0/237670358 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc400dd40 con 0x7f0bcc0b4370 2023-11-24T06:20:42.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bcb7fe700 1 -- 172.21.15.89:0/237670358 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc400b310 con 0x7f0bcc0b4370 2023-11-24T06:20:42.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bd2dec700 1 -- 172.21.15.89:0/237670358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc0b4370 msgr2=0x7f0bcc15b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/237670358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc0b4370 0x7f0bcc15b800 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d150 tx=0x7f0bc400d460 comp rx=0 tx=0).stop 2023-11-24T06:20:42.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bd2dec700 1 -- 172.21.15.89:0/237670358 shutdown_connections 2023-11-24T06:20:42.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/237670358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc0b4370 0x7f0bcc15b800 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/237670358 >> v2:172.21.15.123:3300/0 conn(0x7f0bcc1587f0 0x7f0bcc06aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bd2dec700 1 -- 172.21.15.89:0/237670358 >> 172.21.15.89:0/237670358 conn(0x7f0bcc0d0bf0 msgr2=0x7f0bcc0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:42.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bd2dec700 1 -- 172.21.15.89:0/237670358 shutdown_connections 2023-11-24T06:20:42.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.621+0000 7f0bd2dec700 1 -- 172.21.15.89:0/237670358 wait complete. 2023-11-24T06:20:42.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.622+0000 7f0bd2dec700 1 Processor -- start 2023-11-24T06:20:42.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.622+0000 7f0bd2dec700 1 -- start start 2023-11-24T06:20:42.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd2dec700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc0b4370 0x7f0bcc074260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd2dec700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc1587f0 0x7f0bcc1f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd2dec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc1f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd2dec700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc1e3930 con 0x7f0bcc15b370 2023-11-24T06:20:42.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd2dec700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc1559f0 con 0x7f0bcc0b4370 2023-11-24T06:20:42.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd2dec700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc155cd0 con 0x7f0bcc1587f0 2023-11-24T06:20:42.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd1389700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc1f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd1389700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc1f2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38470/0 (socket says 172.21.15.89:38470) 2023-11-24T06:20:42.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.623+0000 7f0bd1389700 1 -- 172.21.15.89:0/1882797000 learned_addr learned my addr 172.21.15.89:0/1882797000 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:42.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd1389700 1 -- 172.21.15.89:0/1882797000 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc1587f0 msgr2=0x7f0bcc1f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd1389700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc1587f0 0x7f0bcc1f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd1389700 1 -- 172.21.15.89:0/1882797000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc0b4370 msgr2=0x7f0bcc074260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd0b88700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc0b4370 0x7f0bcc074260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd1389700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc0b4370 0x7f0bcc074260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd1389700 1 -- 172.21.15.89:0/1882797000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc400bd40 con 0x7f0bcc15b370 2023-11-24T06:20:42.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd0b88700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc0b4370 0x7f0bcc074260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:42.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd1389700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc1f2d10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4017040 tx=0x7f0bc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bc97fa700 1 -- 172.21.15.89:0/1882797000 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4009a10 con 0x7f0bcc15b370 2023-11-24T06:20:42.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bc97fa700 1 -- 172.21.15.89:0/1882797000 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4009b70 con 0x7f0bcc15b370 2023-11-24T06:20:42.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc155fb0 con 0x7f0bcc15b370 2023-11-24T06:20:42.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bc97fa700 1 -- 172.21.15.89:0/1882797000 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4004030 con 0x7f0bcc15b370 2023-11-24T06:20:42.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.624+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc0747a0 con 0x7f0bcc15b370 2023-11-24T06:20:42.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.625+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc074930 con 0x7f0bcc15b370 2023-11-24T06:20:42.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.625+0000 7f0bc97fa700 1 -- 172.21.15.89:0/1882797000 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f0bc4004190 con 0x7f0bcc15b370 2023-11-24T06:20:42.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.626+0000 7f0bc97fa700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bb006af60 0x7f0bb006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.626+0000 7f0bc97fa700 1 -- 172.21.15.89:0/1882797000 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bc408ceb0 con 0x7f0bcc15b370 2023-11-24T06:20:42.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.626+0000 7f0bd0b88700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bb006af60 0x7f0bb006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.626+0000 7f0bd0b88700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bb006af60 0x7f0bb006d420 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009a00 tx=0x7f0bc0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.630+0000 7f0bc97fa700 1 -- 172.21.15.89:0/1882797000 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bc4057ef0 con 0x7f0bcc15b370 2023-11-24T06:20:42.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.821+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0bcc068fb0 con 0x7f0bcc15b370 2023-11-24T06:20:42.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.822+0000 7f0bc97fa700 1 -- 172.21.15.89:0/1882797000 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0bc4019020 con 0x7f0bcc15b370 2023-11-24T06:20:42.825 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:20:42.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bb006af60 msgr2=0x7f0bb006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bb006af60 0x7f0bb006d420 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009a00 tx=0x7f0bc0008040 comp rx=0 tx=0).stop 2023-11-24T06:20:42.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 msgr2=0x7f0bcc1f2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc1f2d10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4017040 tx=0x7f0bc4004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 shutdown_connections 2023-11-24T06:20:42.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bcc15b370 0x7f0bcc1f2d10 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0bb006af60 0x7f0bb006d420 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bcc0b4370 0x7f0bcc074260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 --2- 172.21.15.89:0/1882797000 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bcc1587f0 0x7f0bcc1f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 >> 172.21.15.89:0/1882797000 conn(0x7f0bcc0d0bf0 msgr2=0x7f0bcc0d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:42.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 shutdown_connections 2023-11-24T06:20:42.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.824+0000 7f0bd2dec700 1 -- 172.21.15.89:0/1882797000 wait complete. 2023-11-24T06:20:42.839 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:20:42.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.959+0000 7f0bb9aa5700 1 Processor -- start 2023-11-24T06:20:42.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.959+0000 7f0bb9aa5700 1 -- start start 2023-11-24T06:20:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb9aa5700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0bb415e160 0x7f0bb40b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb9aa5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bb40746a0 0x7f0bb406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb9aa5700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb40b3c60 con 0x7f0bb40afc90 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb9aa5700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb40b4500 con 0x7f0bb40746a0 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb9aa5700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb40b58c0 con 0x7f0bb415e160 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb3fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bb40746a0 0x7f0bb406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb2ffd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0bb40afc90 0x7f0bb4073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41980/0 (socket says 172.21.15.89:41980) 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb2ffd700 1 -- 172.21.15.89:0/1711832496 learned_addr learned my addr 172.21.15.89:0/1711832496 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb37fe700 1 --2- 172.21.15.89:0/1711832496 >> v2:172.21.15.123:3300/0 conn(0x7f0bb415e160 0x7f0bb40b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb3fff700 1 -- 172.21.15.89:0/1711832496 >> v2:172.21.15.123:3300/0 conn(0x7f0bb415e160 msgr2=0x7f0bb40b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb3fff700 1 --2- 172.21.15.89:0/1711832496 >> v2:172.21.15.123:3300/0 conn(0x7f0bb415e160 0x7f0bb40b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb3fff700 1 -- 172.21.15.89:0/1711832496 >> v1:172.21.15.89:6789/0 conn(0x7f0bb40afc90 legacy=0x7f0bb4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb27fc700 1 -- 172.21.15.89:0/1711832496 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510606828 0 0) 0x7f0bb40b3c60 con 0x7f0bb40afc90 2023-11-24T06:20:42.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb3fff700 1 -- 172.21.15.89:0/1711832496 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb40b0140 con 0x7f0bb40746a0 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.960+0000 7f0bb3fff700 1 --2- 172.21.15.89:0/1711832496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb406d930 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0bb409efa0 tx=0x7f0bac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b0a5efbffbd961 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb27fc700 1 -- 172.21.15.89:0/1711832496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00e070 con 0x7f0bb40746a0 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb27fc700 1 -- 172.21.15.89:0/1711832496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac00bb20 con 0x7f0bb40746a0 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb27fc700 1 -- 172.21.15.89:0/1711832496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac012430 con 0x7f0bb40746a0 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/1711832496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 msgr2=0x7f0bb406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/1711832496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb406d930 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0bb409efa0 tx=0x7f0bac00b330 comp rx=0 tx=0).stop 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/1711832496 shutdown_connections 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/1711832496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb406d930 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/1711832496 >> v2:172.21.15.123:3300/0 conn(0x7f0bb415e160 0x7f0bb40b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/1711832496 >> 172.21.15.89:0/1711832496 conn(0x7f0bb40d0bf0 msgr2=0x7f0bb40d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:42.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/1711832496 shutdown_connections 2023-11-24T06:20:42.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.961+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/1711832496 wait complete. 2023-11-24T06:20:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.962+0000 7f0bb9aa5700 1 Processor -- start 2023-11-24T06:20:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb9aa5700 1 -- start start 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb9aa5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb4154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb9aa5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb40afc90 0x7f0bb41f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb9aa5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb415e160 0x7f0bb41f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb9aa5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4154800 con 0x7f0bb40746a0 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb9aa5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4154ae0 con 0x7f0bb40afc90 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb9aa5700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40b0540 con 0x7f0bb415e160 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb4154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb4154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38480/0 (socket says 172.21.15.89:38480) 2023-11-24T06:20:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.963+0000 7f0bb37fe700 1 -- 172.21.15.89:0/751464869 learned_addr learned my addr 172.21.15.89:0/751464869 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb3fff700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb415e160 0x7f0bb41f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb2ffd700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb40afc90 0x7f0bb41f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb37fe700 1 -- 172.21.15.89:0/751464869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb415e160 msgr2=0x7f0bb41f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb37fe700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb415e160 0x7f0bb41f4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb37fe700 1 -- 172.21.15.89:0/751464869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb40afc90 msgr2=0x7f0bb41f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb37fe700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb40afc90 0x7f0bb41f5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb37fe700 1 -- 172.21.15.89:0/751464869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac009a70 con 0x7f0bb40746a0 2023-11-24T06:20:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb37fe700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb4154220 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800e850 tx=0x7f0ba800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0b9bfff700 1 -- 172.21.15.89:0/751464869 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80095b0 con 0x7f0bb40746a0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0b9bfff700 1 -- 172.21.15.89:0/751464869 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba800ee50 con 0x7f0bb40746a0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb40b0820 con 0x7f0bb40746a0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0b9bfff700 1 -- 172.21.15.89:0/751464869 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba800ca70 con 0x7f0bb40746a0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.964+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb40aeb80 con 0x7f0bb40746a0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.965+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb40b03a0 con 0x7f0bb40746a0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.965+0000 7f0b9bfff700 1 -- 172.21.15.89:0/751464869 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f0ba800cbd0 con 0x7f0bb40746a0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.966+0000 7f0b9bfff700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0ba006af60 0x7f0ba006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.966+0000 7f0b9bfff700 1 -- 172.21.15.89:0/751464869 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ba80895a0 con 0x7f0bb40746a0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.966+0000 7f0bb2ffd700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0ba006af60 0x7f0ba006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.966+0000 7f0bb2ffd700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0ba006af60 0x7f0ba006d420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0bb406ee90 tx=0x7f0ba4007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:42.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:42.970+0000 7f0b9bfff700 1 -- 172.21.15.89:0/751464869 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bb40b03a0 con 0x7f0bb40746a0 2023-11-24T06:20:43.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.162+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0bb40b03a0 con 0x7f0bb40746a0 2023-11-24T06:20:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.162+0000 7f0b9bfff700 1 -- 172.21.15.89:0/751464869 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0bb40b03a0 con 0x7f0bb40746a0 2023-11-24T06:20:43.166 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:20:43.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.164+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0ba006af60 msgr2=0x7f0ba006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.164+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0ba006af60 0x7f0ba006d420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0bb406ee90 tx=0x7f0ba4007be0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.164+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 msgr2=0x7f0bb4154220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.164+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb4154220 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800e850 tx=0x7f0ba800ec10 comp rx=0 tx=0).stop 2023-11-24T06:20:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.165+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 shutdown_connections 2023-11-24T06:20:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.165+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb415e160 0x7f0bb41f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.165+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb40746a0 0x7f0bb4154220 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.165+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0ba006af60 0x7f0ba006d420 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.165+0000 7f0bb9aa5700 1 --2- 172.21.15.89:0/751464869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb40afc90 0x7f0bb41f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.165+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 >> 172.21.15.89:0/751464869 conn(0x7f0bb40d0bf0 msgr2=0x7f0bb415d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.165+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 shutdown_connections 2023-11-24T06:20:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.165+0000 7f0bb9aa5700 1 -- 172.21.15.89:0/751464869 wait complete. 2023-11-24T06:20:43.180 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:43.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.298+0000 7f26a0a38700 1 Processor -- start 2023-11-24T06:20:43.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.298+0000 7f26a0a38700 1 -- start start 2023-11-24T06:20:43.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f26a0a38700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f269c15b200 0x7f269c0b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f26a0a38700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269c0cfd40 con 0x7f269c15ae20 2023-11-24T06:20:43.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f26a0a38700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269c0affa0 con 0x7f269c0b1a70 2023-11-24T06:20:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f26a0a38700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0d0020 con 0x7f269c15b200 2023-11-24T06:20:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269a59c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f269c15ae20 0x7f269c0b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41992/0 (socket says 172.21.15.89:41992) 2023-11-24T06:20:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269a59c700 1 -- 172.21.15.89:0/2939447397 learned_addr learned my addr 172.21.15.89:0/2939447397 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412250013 0 0) 0x7f269c0cfd40 con 0x7f269c15ae20 2023-11-24T06:20:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26840035b0 con 0x7f269c15ae20 2023-11-24T06:20:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f2699d9b700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f269c15b200 0x7f269c0b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2785136781 0 0) 0x7f26840035b0 con 0x7f269c15ae20 2023-11-24T06:20:43.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f269c15b200 msgr2=0x7f269c0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:43.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f269c15b200 0x7f269c0b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 >> v1:172.21.15.123:6789/0 conn(0x7f269c0b1a70 legacy=0x7f269c164b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c0b0280 con 0x7f269c15ae20 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f268c001670 con 0x7f269c15ae20 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 >> v1:172.21.15.89:6789/0 conn(0x7f269c15ae20 legacy=0x7f269c0b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2684003a30 0x7f2684003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f269c15b200 0x7f2684004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2684004d90 0x7f2684005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0d0020 con 0x7f2684003a30 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268400baf0 con 0x7f2684004d90 2023-11-24T06:20:43.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268400bcb0 con 0x7f269c15b200 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.299+0000 7f269a59c700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2684003a30 0x7f2684003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f2699d9b700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f269c15b200 0x7f2684004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269ad9d700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2684004d90 0x7f2684005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269a59c700 1 -- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f269c15b200 msgr2=0x7f2684004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269a59c700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f269c15b200 0x7f2684004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269a59c700 1 -- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2684004d90 msgr2=0x7f2684005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269a59c700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2684004d90 0x7f2684005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269a59c700 1 -- 172.21.15.89:0/2939447397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26840035b0 con 0x7f2684003a30 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269ad9d700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2684004d90 0x7f2684005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269a59c700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2684003a30 0x7f2684003e50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f268c00c210 tx=0x7f2684005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42c94e3c51629da server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c007c10 con 0x7f2684003a30 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c00cdb0 con 0x7f2684003a30 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f269959a700 1 -- 172.21.15.89:0/2939447397 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c005ba0 con 0x7f2684003a30 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f26a0a38700 1 -- 172.21.15.89:0/2939447397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2684003a30 msgr2=0x7f2684003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.300+0000 7f26a0a38700 1 --2- 172.21.15.89:0/2939447397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2684003a30 0x7f2684003e50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f268c00c210 tx=0x7f2684005880 comp rx=0 tx=0).stop 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.301+0000 7f269ad9d700 1 -- 172.21.15.89:0/2939447397 reap_dead start 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.301+0000 7f26a0a38700 1 -- 172.21.15.89:0/2939447397 shutdown_connections 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.301+0000 7f26a0a38700 1 -- 172.21.15.89:0/2939447397 >> 172.21.15.89:0/2939447397 conn(0x7f269c0b57b0 msgr2=0x7f269c0b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.301+0000 7f26a0a38700 1 -- 172.21.15.89:0/2939447397 shutdown_connections 2023-11-24T06:20:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.301+0000 7f26a0a38700 1 -- 172.21.15.89:0/2939447397 wait complete. 2023-11-24T06:20:43.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.302+0000 7f26a0a38700 1 Processor -- start 2023-11-24T06:20:43.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.302+0000 7f26a0a38700 1 -- start start 2023-11-24T06:20:43.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f26a0a38700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2684003a30 0x7f269c14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f26a0a38700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f269c09a380 0x7f269c0c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f26a0a38700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f269c09a800 0x7f269c0c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f26a0a38700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c14dce0 con 0x7f269c09a800 2023-11-24T06:20:43.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f26a0a38700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c15c910 con 0x7f269c09a380 2023-11-24T06:20:43.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f26a0a38700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c09d120 con 0x7f2684003a30 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f269c09a800 0x7f269c0c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f269c09a800 0x7f269c0c9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38504/0 (socket says 172.21.15.89:38504) 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 -- 172.21.15.89:0/3778154972 learned_addr learned my addr 172.21.15.89:0/3778154972 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f2699d9b700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f269c09a380 0x7f269c0c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269a59c700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2684003a30 0x7f269c14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 -- 172.21.15.89:0/3778154972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2684003a30 msgr2=0x7f269c14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2684003a30 0x7f269c14d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 -- 172.21.15.89:0/3778154972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f269c09a380 msgr2=0x7f269c0c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f269c09a380 0x7f269c0c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 -- 172.21.15.89:0/3778154972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c09d400 con 0x7f269c09a800 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f269ad9d700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f269c09a800 0x7f269c0c9b50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f269c0717a0 tx=0x7f269400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f268affd700 1 -- 172.21.15.89:0/3778154972 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694003c10 con 0x7f269c09a800 2023-11-24T06:20:43.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f268affd700 1 -- 172.21.15.89:0/3778154972 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2694003d90 con 0x7f269c09a800 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c09d6e0 con 0x7f269c09a800 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f268affd700 1 -- 172.21.15.89:0/3778154972 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694005310 con 0x7f269c09a800 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.303+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c09dca0 con 0x7f269c09a800 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.304+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c067f60 con 0x7f269c09a800 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.305+0000 7f268affd700 1 -- 172.21.15.89:0/3778154972 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f2694005470 con 0x7f269c09a800 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.305+0000 7f268affd700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f267c06af60 0x7f267c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.305+0000 7f269a59c700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f267c06af60 0x7f267c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.305+0000 7f268affd700 1 -- 172.21.15.89:0/3778154972 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26940895d0 con 0x7f269c09a800 2023-11-24T06:20:43.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.306+0000 7f269a59c700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f267c06af60 0x7f267c06d420 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f268c001930 tx=0x7f268c007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:43.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.310+0000 7f268affd700 1 -- 172.21.15.89:0/3778154972 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2694054720 con 0x7f269c09a800 2023-11-24T06:20:43.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.544+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f269c09dec0 con 0x7f269c09a800 2023-11-24T06:20:43.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.544+0000 7f268affd700 1 -- 172.21.15.89:0/3778154972 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f2694001b50 con 0x7f269c09a800 2023-11-24T06:20:43.547 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:43.547 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2011089843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2011089843},{"type":"v1","addr":"172.21.15.89:6836","nonce":2011089843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4955,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/2182962276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2182962276},{"type":"v1","addr":"172.21.15.123:6834","nonce":2182962276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2982219407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2982219407},{"type":"v1","addr":"172.21.15.123:6835","nonce":2982219407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:22.026873+0000","modified":"2023-11-24T06:20:25.053779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5577":{"gid":5577,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1341821445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1341821445},{"type":"v1","addr":"172.21.15.89:6837","nonce":1341821445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T06:20:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f267c06af60 msgr2=0x7f267c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f267c06af60 0x7f267c06d420 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f268c001930 tx=0x7f268c007000 comp rx=0 tx=0).stop 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f269c09a800 msgr2=0x7f269c0c9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f269c09a800 0x7f269c0c9b50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f269c0717a0 tx=0x7f269400ee70 comp rx=0 tx=0).stop 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 shutdown_connections 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f269c09a800 0x7f269c0c9b50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2684003a30 0x7f269c14d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f267c06af60 0x7f267c06d420 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 --2- 172.21.15.89:0/3778154972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f269c09a380 0x7f269c0c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 >> 172.21.15.89:0/3778154972 conn(0x7f269c0b57b0 msgr2=0x7f269c0b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 shutdown_connections 2023-11-24T06:20:43.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.546+0000 7f26a0a38700 1 -- 172.21.15.89:0/3778154972 wait complete. 2023-11-24T06:20:43.551 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-11-24T06:20:43.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:43.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.679+0000 7f67c01df700 1 Processor -- start 2023-11-24T06:20:43.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.679+0000 7f67c01df700 1 -- start start 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67c01df700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f67b81446a0 0x7f67b80b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67c01df700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b80b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67c01df700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b80af950 con 0x7f67b80adf00 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67c01df700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b8144a80 con 0x7f67b81446a0 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67c01df700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b80afc30 con 0x7f67b806c710 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67be77c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b80b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67bdf7b700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f67b81446a0 0x7f67b80b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67be77c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b80b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:47386/0 (socket says 172.21.15.89:47386) 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67bd77a700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f67b80adf00 0x7f67b806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:45834/0 (socket says 172.21.15.89:45834) 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67be77c700 1 -- 172.21.15.89:0/2832663752 learned_addr learned my addr 172.21.15.89:0/2832663752 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67be77c700 1 -- 172.21.15.89:0/2832663752 >> v2:172.21.15.123:3300/0 conn(0x7f67b81446a0 msgr2=0x7f67b80b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:43.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67be77c700 1 --2- 172.21.15.89:0/2832663752 >> v2:172.21.15.123:3300/0 conn(0x7f67b81446a0 0x7f67b80b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67be77c700 1 -- 172.21.15.89:0/2832663752 >> v1:172.21.15.123:6789/0 conn(0x7f67b80adf00 legacy=0x7f67b806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.680+0000 7f67be77c700 1 -- 172.21.15.89:0/2832663752 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b80aff10 con 0x7f67b806c710 2023-11-24T06:20:43.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67be77c700 1 --2- 172.21.15.89:0/2832663752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b80b3d40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f67b000d150 tx=0x7f67b000d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49c51c59307d0efe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:43.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67bcf79700 1 -- 172.21.15.89:0/2832663752 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b0010070 con 0x7f67b806c710 2023-11-24T06:20:43.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67bcf79700 1 -- 172.21.15.89:0/2832663752 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b000dd10 con 0x7f67b806c710 2023-11-24T06:20:43.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67bcf79700 1 -- 172.21.15.89:0/2832663752 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b000b310 con 0x7f67b806c710 2023-11-24T06:20:43.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67c01df700 1 -- 172.21.15.89:0/2832663752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 msgr2=0x7f67b80b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67c01df700 1 --2- 172.21.15.89:0/2832663752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b80b3d40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f67b000d150 tx=0x7f67b000d460 comp rx=0 tx=0).stop 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67c01df700 1 -- 172.21.15.89:0/2832663752 shutdown_connections 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67c01df700 1 --2- 172.21.15.89:0/2832663752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b80b3d40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67c01df700 1 --2- 172.21.15.89:0/2832663752 >> v2:172.21.15.123:3300/0 conn(0x7f67b81446a0 0x7f67b80b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67c01df700 1 -- 172.21.15.89:0/2832663752 >> 172.21.15.89:0/2832663752 conn(0x7f67b8071b90 msgr2=0x7f67b8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67c01df700 1 -- 172.21.15.89:0/2832663752 shutdown_connections 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.681+0000 7f67c01df700 1 -- 172.21.15.89:0/2832663752 wait complete. 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.682+0000 7f67c01df700 1 Processor -- start 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67c01df700 1 -- start start 2023-11-24T06:20:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67c01df700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b8156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67c01df700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b80adf00 0x7f67b813c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67c01df700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67b81446a0 0x7f67b8143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67c01df700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b81e4aa0 con 0x7f67b80adf00 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67c01df700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b80b4ae0 con 0x7f67b81446a0 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67c01df700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b81e4ec0 con 0x7f67b806c710 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67bd77a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b80adf00 0x7f67b813c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67bd77a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b80adf00 0x7f67b813c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38514/0 (socket says 172.21.15.89:38514) 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.683+0000 7f67bd77a700 1 -- 172.21.15.89:0/2003361375 learned_addr learned my addr 172.21.15.89:0/2003361375 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67bdf7b700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b8156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67be77c700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67b81446a0 0x7f67b8143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67bd77a700 1 -- 172.21.15.89:0/2003361375 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 msgr2=0x7f67b8156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67bd77a700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b8156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67bd77a700 1 -- 172.21.15.89:0/2003361375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67b81446a0 msgr2=0x7f67b8143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67bd77a700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67b81446a0 0x7f67b8143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67bd77a700 1 -- 172.21.15.89:0/2003361375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b000bd40 con 0x7f67b80adf00 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67bd77a700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b80adf00 0x7f67b813c020 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f67a800c400 tx=0x7f67a800c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67ae7fc700 1 -- 172.21.15.89:0/2003361375 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8005cb0 con 0x7f67b80adf00 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67ae7fc700 1 -- 172.21.15.89:0/2003361375 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a8005e10 con 0x7f67b80adf00 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b81f4d00 con 0x7f67b80adf00 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67ae7fc700 1 -- 172.21.15.89:0/2003361375 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a80072d0 con 0x7f67b80adf00 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.684+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b81f4a20 con 0x7f67b80adf00 2023-11-24T06:20:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.685+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b8157f00 con 0x7f67b80adf00 2023-11-24T06:20:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.685+0000 7f67ae7fc700 1 -- 172.21.15.89:0/2003361375 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f67a8007430 con 0x7f67b80adf00 2023-11-24T06:20:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.686+0000 7f67ae7fc700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f679c06af60 0x7f679c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.686+0000 7f67bdf7b700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f679c06af60 0x7f679c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.686+0000 7f67ae7fc700 1 -- 172.21.15.89:0/2003361375 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67a80887b0 con 0x7f67b80adf00 2023-11-24T06:20:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.686+0000 7f67bdf7b700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f679c06af60 0x7f679c06d420 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009b40 tx=0x7f67b4009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:43.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.690+0000 7f67ae7fc700 1 -- 172.21.15.89:0/2003361375 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67a8053920 con 0x7f67b80adf00 2023-11-24T06:20:43.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.882+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67b80753f0 con 0x7f67b80adf00 2023-11-24T06:20:43.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.883+0000 7f67ae7fc700 1 -- 172.21.15.89:0/2003361375 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16529 (secure 0 0 0) 0x7f67a8017090 con 0x7f67b80adf00 2023-11-24T06:20:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:43.887 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":53,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:23.644194+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:20.413276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:20.978712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.885+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f679c06af60 msgr2=0x7f679c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.885+0000 7f67c01df700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f679c06af60 0x7f679c06d420 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009b40 tx=0x7f67b4009210 comp rx=0 tx=0).stop 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b80adf00 msgr2=0x7f67b813c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b80adf00 0x7f67b813c020 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f67a800c400 tx=0x7f67a800c710 comp rx=0 tx=0).stop 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 shutdown_connections 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f679c06af60 0x7f679c06d420 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67b81446a0 0x7f67b8143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67b806c710 0x7f67b8156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 --2- 172.21.15.89:0/2003361375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b80adf00 0x7f67b813c020 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:43.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 >> 172.21.15.89:0/2003361375 conn(0x7f67b8071b90 msgr2=0x7f67b815e710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:43.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 shutdown_connections 2023-11-24T06:20:43.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:43.886+0000 7f67c01df700 1 -- 172.21.15.89:0/2003361375 wait complete. 2023-11-24T06:20:43.901 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T06:20:43.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:44.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.019+0000 7f2c97c88700 1 Processor -- start 2023-11-24T06:20:44.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.020+0000 7f2c97c88700 1 -- start start 2023-11-24T06:20:44.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.020+0000 7f2c97c88700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c90073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.020+0000 7f2c97c88700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2c9006fec0 0x7f2c900744d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.020+0000 7f2c97c88700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c900af090 con 0x7f2c900afc10 2023-11-24T06:20:44.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.020+0000 7f2c97c88700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9006dda0 con 0x7f2c9006fec0 2023-11-24T06:20:44.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.020+0000 7f2c97c88700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9006e060 con 0x7f2c9006fa40 2023-11-24T06:20:44.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.020+0000 7f2c96225700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2c9006fec0 0x7f2c900744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.020+0000 7f2c95a24700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2c900afc10 0x7f2c9015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42028/0 (socket says 172.21.15.89:42028) 2023-11-24T06:20:44.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c95a24700 1 -- 172.21.15.89:0/4251172551 learned_addr learned my addr 172.21.15.89:0/4251172551 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:44.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c95223700 1 --2- 172.21.15.89:0/4251172551 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c90073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c96225700 1 -- 172.21.15.89:0/4251172551 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 msgr2=0x7f2c90073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:44.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c96225700 1 --2- 172.21.15.89:0/4251172551 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c90073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c96225700 1 -- 172.21.15.89:0/4251172551 >> v1:172.21.15.89:6789/0 conn(0x7f2c900afc10 legacy=0x7f2c9015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c94a22700 1 -- 172.21.15.89:0/4251172551 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461988123 0 0) 0x7f2c900af090 con 0x7f2c900afc10 2023-11-24T06:20:44.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c96225700 1 -- 172.21.15.89:0/4251172551 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9006ec70 con 0x7f2c9006fec0 2023-11-24T06:20:44.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c96225700 1 --2- 172.21.15.89:0/4251172551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c900744d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2c88009d90 tx=0x7f2c8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c4317e6e0b5b4e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c94a22700 1 -- 172.21.15.89:0/4251172551 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8800e070 con 0x7f2c9006fec0 2023-11-24T06:20:44.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c94a22700 1 -- 172.21.15.89:0/4251172551 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8800bbd0 con 0x7f2c9006fec0 2023-11-24T06:20:44.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.021+0000 7f2c94a22700 1 -- 172.21.15.89:0/4251172551 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c88012430 con 0x7f2c9006fec0 2023-11-24T06:20:44.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.022+0000 7f2c97c88700 1 -- 172.21.15.89:0/4251172551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 msgr2=0x7f2c900744d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.022+0000 7f2c97c88700 1 --2- 172.21.15.89:0/4251172551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c900744d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2c88009d90 tx=0x7f2c8800b330 comp rx=0 tx=0).stop 2023-11-24T06:20:44.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.022+0000 7f2c97c88700 1 -- 172.21.15.89:0/4251172551 shutdown_connections 2023-11-24T06:20:44.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.022+0000 7f2c97c88700 1 --2- 172.21.15.89:0/4251172551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c900744d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.022+0000 7f2c97c88700 1 --2- 172.21.15.89:0/4251172551 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c90073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.022+0000 7f2c97c88700 1 -- 172.21.15.89:0/4251172551 >> 172.21.15.89:0/4251172551 conn(0x7f2c9006aff0 msgr2=0x7f2c9006b400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:44.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.022+0000 7f2c97c88700 1 -- 172.21.15.89:0/4251172551 shutdown_connections 2023-11-24T06:20:44.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.022+0000 7f2c97c88700 1 -- 172.21.15.89:0/4251172551 wait complete. 2023-11-24T06:20:44.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.023+0000 7f2c97c88700 1 Processor -- start 2023-11-24T06:20:44.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.023+0000 7f2c97c88700 1 -- start start 2023-11-24T06:20:44.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c97c88700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c9015b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c97c88700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c9015d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c97c88700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c900afc10 0x7f2c9015d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c97c88700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9015dce0 con 0x7f2c9006fec0 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c97c88700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c90158e60 con 0x7f2c900afc10 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c97c88700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c90159140 con 0x7f2c9006fa40 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c95223700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c9015d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c95223700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c9015d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38528/0 (socket says 172.21.15.89:38528) 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c95223700 1 -- 172.21.15.89:0/32966398 learned_addr learned my addr 172.21.15.89:0/32966398 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.024+0000 7f2c95a24700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c9015b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c96225700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c900afc10 0x7f2c9015d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c95223700 1 -- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 msgr2=0x7f2c9015b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c95223700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c9015b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c95223700 1 -- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c900afc10 msgr2=0x7f2c9015d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c95223700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c900afc10 0x7f2c9015d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c95223700 1 -- 172.21.15.89:0/32966398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c88009a70 con 0x7f2c9006fec0 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c95a24700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c9015b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c95223700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c9015d260 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000e810 tx=0x7f2c8000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c867fc700 1 -- 172.21.15.89:0/32966398 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c80013610 con 0x7f2c9006fec0 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c867fc700 1 -- 172.21.15.89:0/32966398 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c80013770 con 0x7f2c9006fec0 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c867fc700 1 -- 172.21.15.89:0/32966398 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c80013b20 con 0x7f2c9006fec0 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c90159420 con 0x7f2c9006fec0 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.025+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c901599e0 con 0x7f2c9006fec0 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.026+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c900af0c0 con 0x7f2c9006fec0 2023-11-24T06:20:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.026+0000 7f2c867fc700 1 -- 172.21.15.89:0/32966398 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f2c80013c80 con 0x7f2c9006fec0 2023-11-24T06:20:44.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.027+0000 7f2c867fc700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2c7406af60 0x7f2c7406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.027+0000 7f2c867fc700 1 -- 172.21.15.89:0/32966398 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c800894f0 con 0x7f2c9006fec0 2023-11-24T06:20:44.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.027+0000 7f2c95a24700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2c7406af60 0x7f2c7406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.027+0000 7f2c95a24700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2c7406af60 0x7f2c7406d420 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c000f60 tx=0x7f2c8c007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.031+0000 7f2c867fc700 1 -- 172.21.15.89:0/32966398 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c80054420 con 0x7f2c9006fec0 2023-11-24T06:20:44.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.265+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c90068f10 con 0x7f2c9006fec0 2023-11-24T06:20:44.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.265+0000 7f2c867fc700 1 -- 172.21.15.89:0/32966398 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f2c80057a40 con 0x7f2c9006fec0 2023-11-24T06:20:44.268 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:44.268 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2011089843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2011089843},{"type":"v1","addr":"172.21.15.89:6836","nonce":2011089843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4955,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/2182962276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2182962276},{"type":"v1","addr":"172.21.15.123:6834","nonce":2182962276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2982219407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2982219407},{"type":"v1","addr":"172.21.15.123:6835","nonce":2982219407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:22.026873+0000","modified":"2023-11-24T06:20:25.053779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5577":{"gid":5577,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1341821445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1341821445},{"type":"v1","addr":"172.21.15.89:6837","nonce":1341821445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T06:20:44.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2c7406af60 msgr2=0x7f2c7406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2c7406af60 0x7f2c7406d420 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c000f60 tx=0x7f2c8c007d30 comp rx=0 tx=0).stop 2023-11-24T06:20:44.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 msgr2=0x7f2c9015d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c9015d260 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000e810 tx=0x7f2c8000ebd0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 shutdown_connections 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2c7406af60 0x7f2c7406d420 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c900afc10 0x7f2c9015d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c9006fa40 0x7f2c9015b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 --2- 172.21.15.89:0/32966398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c9006fec0 0x7f2c9015d260 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 >> 172.21.15.89:0/32966398 conn(0x7f2c9006aff0 msgr2=0x7f2c9015f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 shutdown_connections 2023-11-24T06:20:44.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.267+0000 7f2c97c88700 1 -- 172.21.15.89:0/32966398 wait complete. 2023-11-24T06:20:44.272 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-11-24T06:20:44.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.400+0000 7efc6be12700 1 Processor -- start 2023-11-24T06:20:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.400+0000 7efc6be12700 1 -- start start 2023-11-24T06:20:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc6be12700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efc6415bdd0 0x7efc641604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc6be12700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc6be12700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc640b2030 con 0x7efc640d1c70 2023-11-24T06:20:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc6be12700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc640b2f50 con 0x7efc6415bdd0 2023-11-24T06:20:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc6be12700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc640b22d0 con 0x7efc640b3370 2023-11-24T06:20:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc693ad700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc69bae700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efc6415bdd0 0x7efc641604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc693ad700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:47410/0 (socket says 172.21.15.89:47410) 2023-11-24T06:20:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc693ad700 1 -- 172.21.15.89:0/1957110290 learned_addr learned my addr 172.21.15.89:0/1957110290 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc6a3af700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7efc640d1c70 0x7efc640ce6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:45864/0 (socket says 172.21.15.89:45864) 2023-11-24T06:20:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.401+0000 7efc69bae700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efc6415bdd0 0x7efc641604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:53646/0 (socket says 172.21.15.89:53646) 2023-11-24T06:20:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc69bae700 1 -- 172.21.15.89:0/1957110290 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 msgr2=0x7efc640b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc69bae700 1 --2- 172.21.15.89:0/1957110290 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc69bae700 1 -- 172.21.15.89:0/1957110290 >> v1:172.21.15.123:6789/0 conn(0x7efc640d1c70 legacy=0x7efc640ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc69bae700 1 -- 172.21.15.89:0/1957110290 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc640b25b0 con 0x7efc6415bdd0 2023-11-24T06:20:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc68bac700 1 -- 172.21.15.89:0/1957110290 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691153247 0 0) 0x7efc640b2030 con 0x7efc640d1c70 2023-11-24T06:20:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc693ad700 1 --2- 172.21.15.89:0/1957110290 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:20:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc69bae700 1 --2- 172.21.15.89:0/1957110290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 0x7efc641604e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc54009840 tx=0x7efc54009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5c5170d45c707fc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc68bac700 1 -- 172.21.15.89:0/1957110290 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54011830 con 0x7efc6415bdd0 2023-11-24T06:20:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc68bac700 1 -- 172.21.15.89:0/1957110290 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc54011990 con 0x7efc6415bdd0 2023-11-24T06:20:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.402+0000 7efc68bac700 1 -- 172.21.15.89:0/1957110290 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54011d20 con 0x7efc6415bdd0 2023-11-24T06:20:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.403+0000 7efc6be12700 1 -- 172.21.15.89:0/1957110290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 msgr2=0x7efc641604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.403+0000 7efc6be12700 1 --2- 172.21.15.89:0/1957110290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 0x7efc641604e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc54009840 tx=0x7efc54009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.403+0000 7efc6be12700 1 -- 172.21.15.89:0/1957110290 shutdown_connections 2023-11-24T06:20:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.403+0000 7efc6be12700 1 --2- 172.21.15.89:0/1957110290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 0x7efc641604e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.403+0000 7efc6be12700 1 --2- 172.21.15.89:0/1957110290 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.403+0000 7efc6be12700 1 -- 172.21.15.89:0/1957110290 >> 172.21.15.89:0/1957110290 conn(0x7efc640731a0 msgr2=0x7efc6409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.403+0000 7efc6be12700 1 -- 172.21.15.89:0/1957110290 shutdown_connections 2023-11-24T06:20:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.403+0000 7efc6be12700 1 -- 172.21.15.89:0/1957110290 wait complete. 2023-11-24T06:20:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.404+0000 7efc6be12700 1 Processor -- start 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.404+0000 7efc6be12700 1 -- start start 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc6be12700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc6be12700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc640d1c70 0x7efc6414cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc6be12700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 0x7efc640c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc6be12700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc640cb100 con 0x7efc640d1c70 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc6be12700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc640ca480 con 0x7efc6415bdd0 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc6be12700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc640ca760 con 0x7efc640b3370 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc640d1c70 0x7efc6414cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc640d1c70 0x7efc6414cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38542/0 (socket says 172.21.15.89:38542) 2023-11-24T06:20:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 -- 172.21.15.89:0/4107245204 learned_addr learned my addr 172.21.15.89:0/4107245204 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc6a3af700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 0x7efc640c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc69bae700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 -- 172.21.15.89:0/4107245204 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 msgr2=0x7efc640cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640cb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 -- 172.21.15.89:0/4107245204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 msgr2=0x7efc640c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 0x7efc640c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 -- 172.21.15.89:0/4107245204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60009550 con 0x7efc640d1c70 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc693ad700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc640d1c70 0x7efc6414cf90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7efc6000ebe0 tx=0x7efc60009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc5a7fc700 1 -- 172.21.15.89:0/4107245204 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6000c8b0 con 0x7efc640d1c70 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc5a7fc700 1 -- 172.21.15.89:0/4107245204 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6000ca10 con 0x7efc640d1c70 2023-11-24T06:20:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.405+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc540094a0 con 0x7efc640d1c70 2023-11-24T06:20:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.406+0000 7efc5a7fc700 1 -- 172.21.15.89:0/4107245204 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6000cd80 con 0x7efc640d1c70 2023-11-24T06:20:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.406+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc640cad20 con 0x7efc640d1c70 2023-11-24T06:20:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.406+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc640cb240 con 0x7efc640d1c70 2023-11-24T06:20:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.407+0000 7efc5a7fc700 1 -- 172.21.15.89:0/4107245204 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7efc60013470 con 0x7efc640d1c70 2023-11-24T06:20:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.407+0000 7efc5a7fc700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efc4806af60 0x7efc4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.407+0000 7efc69bae700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efc4806af60 0x7efc4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.407+0000 7efc5a7fc700 1 -- 172.21.15.89:0/4107245204 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc60089710 con 0x7efc640d1c70 2023-11-24T06:20:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.408+0000 7efc69bae700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efc4806af60 0x7efc4806d420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efc54000c00 tx=0x7efc5401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.412+0000 7efc5a7fc700 1 -- 172.21.15.89:0/4107245204 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc640cb240 con 0x7efc640d1c70 2023-11-24T06:20:44.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.645+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc640cb240 con 0x7efc640d1c70 2023-11-24T06:20:44.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.645+0000 7efc5a7fc700 1 -- 172.21.15.89:0/4107245204 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4919 (secure 0 0 0) 0x7efc640cb240 con 0x7efc640d1c70 2023-11-24T06:20:44.649 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:44.649 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2011089843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2011089843},{"type":"v1","addr":"172.21.15.89:6836","nonce":2011089843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4955,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/2182962276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2182962276},{"type":"v1","addr":"172.21.15.123:6834","nonce":2182962276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2982219407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2982219407},{"type":"v1","addr":"172.21.15.123:6835","nonce":2982219407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:22.026873+0000","modified":"2023-11-24T06:20:25.053779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5577":{"gid":5577,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1341821445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1341821445},{"type":"v1","addr":"172.21.15.89:6837","nonce":1341821445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-24T06:20:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.647+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efc4806af60 msgr2=0x7efc4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.647+0000 7efc6be12700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efc4806af60 0x7efc4806d420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efc54000c00 tx=0x7efc5401c040 comp rx=0 tx=0).stop 2023-11-24T06:20:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.647+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc640d1c70 msgr2=0x7efc6414cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.647+0000 7efc6be12700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc640d1c70 0x7efc6414cf90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7efc6000ebe0 tx=0x7efc60009e70 comp rx=0 tx=0).stop 2023-11-24T06:20:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.647+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 shutdown_connections 2023-11-24T06:20:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.647+0000 7efc6be12700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efc4806af60 0x7efc4806d420 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.648+0000 7efc6be12700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6415bdd0 0x7efc640c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.648+0000 7efc6be12700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc640b3370 0x7efc640cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.648+0000 7efc6be12700 1 --2- 172.21.15.89:0/4107245204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc640d1c70 0x7efc6414cf90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.648+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 >> 172.21.15.89:0/4107245204 conn(0x7efc640731a0 msgr2=0x7efc640b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.648+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 shutdown_connections 2023-11-24T06:20:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.648+0000 7efc6be12700 1 -- 172.21.15.89:0/4107245204 wait complete. 2023-11-24T06:20:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-11-24T06:20:44.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:44.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.780+0000 7ff930306700 1 Processor -- start 2023-11-24T06:20:44.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.780+0000 7ff930306700 1 -- start start 2023-11-24T06:20:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff930306700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280adf00 0x7ff92806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff930306700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff92806c540 0x7ff9280b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff930306700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9280af950 con 0x7ff9281446a0 2023-11-24T06:20:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff930306700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff928144a80 con 0x7ff92806c540 2023-11-24T06:20:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff930306700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92806c9b0 con 0x7ff9280adf00 2023-11-24T06:20:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff92e8a3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff92806c540 0x7ff9280b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff92e8a3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff92806c540 0x7ff9280b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38552/0 (socket says 172.21.15.89:38552) 2023-11-24T06:20:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff92e8a3700 1 -- 172.21.15.89:0/3939901156 learned_addr learned my addr 172.21.15.89:0/3939901156 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff92d8a1700 1 --2- 172.21.15.89:0/3939901156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280adf00 0x7ff92806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff92e8a3700 1 -- 172.21.15.89:0/3939901156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280adf00 msgr2=0x7ff92806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff92e8a3700 1 --2- 172.21.15.89:0/3939901156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280adf00 0x7ff92806c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff92e8a3700 1 -- 172.21.15.89:0/3939901156 >> v1:172.21.15.123:6789/0 conn(0x7ff9281446a0 legacy=0x7ff9280b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.781+0000 7ff92e8a3700 1 -- 172.21.15.89:0/3939901156 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9280afc30 con 0x7ff92806c540 2023-11-24T06:20:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff92e8a3700 1 --2- 172.21.15.89:0/3939901156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92806c540 0x7ff9280b3d40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff92000d180 tx=0x7ff92000d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf93b2b13557f32 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff92d0a0700 1 -- 172.21.15.89:0/3939901156 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920010070 con 0x7ff92806c540 2023-11-24T06:20:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff92d0a0700 1 -- 172.21.15.89:0/3939901156 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92000ddb0 con 0x7ff92806c540 2023-11-24T06:20:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff92d0a0700 1 -- 172.21.15.89:0/3939901156 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92000b310 con 0x7ff92806c540 2023-11-24T06:20:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff930306700 1 -- 172.21.15.89:0/3939901156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92806c540 msgr2=0x7ff9280b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff930306700 1 --2- 172.21.15.89:0/3939901156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92806c540 0x7ff9280b3d40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff92000d180 tx=0x7ff92000d490 comp rx=0 tx=0).stop 2023-11-24T06:20:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff930306700 1 -- 172.21.15.89:0/3939901156 shutdown_connections 2023-11-24T06:20:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff930306700 1 --2- 172.21.15.89:0/3939901156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff92806c540 0x7ff9280b3d40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff930306700 1 --2- 172.21.15.89:0/3939901156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280adf00 0x7ff92806c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff930306700 1 -- 172.21.15.89:0/3939901156 >> 172.21.15.89:0/3939901156 conn(0x7ff928071b90 msgr2=0x7ff928071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff930306700 1 -- 172.21.15.89:0/3939901156 shutdown_connections 2023-11-24T06:20:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.782+0000 7ff930306700 1 -- 172.21.15.89:0/3939901156 wait complete. 2023-11-24T06:20:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.783+0000 7ff930306700 1 Processor -- start 2023-11-24T06:20:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.783+0000 7ff930306700 1 -- start start 2023-11-24T06:20:44.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff930306700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff92806c540 0x7ff9280cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff930306700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9280adf00 0x7ff9281f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff930306700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281446a0 0x7ff9281f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff930306700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280cde50 con 0x7ff9280adf00 2023-11-24T06:20:44.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff930306700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280ce130 con 0x7ff9281446a0 2023-11-24T06:20:44.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff930306700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280ce410 con 0x7ff92806c540 2023-11-24T06:20:44.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff92d8a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9280adf00 0x7ff9281f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff92d8a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9280adf00 0x7ff9281f3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38556/0 (socket says 172.21.15.89:38556) 2023-11-24T06:20:44.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff92d8a1700 1 -- 172.21.15.89:0/754027237 learned_addr learned my addr 172.21.15.89:0/754027237 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:44.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff92e8a3700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281446a0 0x7ff9281f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.784+0000 7ff92e0a2700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff92806c540 0x7ff9280cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff92d8a1700 1 -- 172.21.15.89:0/754027237 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff92806c540 msgr2=0x7ff9280cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff92d8a1700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff92806c540 0x7ff9280cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff92d8a1700 1 -- 172.21.15.89:0/754027237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281446a0 msgr2=0x7ff9281f4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff92d8a1700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281446a0 0x7ff9281f4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff92d8a1700 1 -- 172.21.15.89:0/754027237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92000bd40 con 0x7ff9280adf00 2023-11-24T06:20:44.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff92d8a1700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9280adf00 0x7ff9281f3ce0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff92400e7e0 tx=0x7ff92400eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff91effd700 1 -- 172.21.15.89:0/754027237 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92400c070 con 0x7ff9280adf00 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff91effd700 1 -- 172.21.15.89:0/754027237 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff924013610 con 0x7ff9280adf00 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9280ce6f0 con 0x7ff9280adf00 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff91effd700 1 -- 172.21.15.89:0/754027237 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924013980 con 0x7ff9280adf00 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.785+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9280cdb40 con 0x7ff9280adf00 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.786+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9280753f0 con 0x7ff9280adf00 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.786+0000 7ff91effd700 1 -- 172.21.15.89:0/754027237 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ff924013ae0 con 0x7ff9280adf00 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.787+0000 7ff91effd700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff90c06af60 0x7ff90c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.787+0000 7ff91effd700 1 -- 172.21.15.89:0/754027237 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff924089430 con 0x7ff9280adf00 2023-11-24T06:20:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.787+0000 7ff92e0a2700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff90c06af60 0x7ff90c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.787+0000 7ff92e0a2700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff90c06af60 0x7ff90c06d420 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff918007fd0 tx=0x7ff918009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.791+0000 7ff91effd700 1 -- 172.21.15.89:0/754027237 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9240543f0 con 0x7ff9280adf00 2023-11-24T06:20:44.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.981+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff928137d10 con 0x7ff9280adf00 2023-11-24T06:20:44.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.982+0000 7ff91effd700 1 -- 172.21.15.89:0/754027237 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16529 (secure 0 0 0) 0x7ff928137d10 con 0x7ff9280adf00 2023-11-24T06:20:44.985 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:44.985 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":53,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:23.644194+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:20.413276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:20.978712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff90c06af60 msgr2=0x7ff90c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff90c06af60 0x7ff90c06d420 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff918007fd0 tx=0x7ff918009040 comp rx=0 tx=0).stop 2023-11-24T06:20:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9280adf00 msgr2=0x7ff9281f3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9280adf00 0x7ff9281f3ce0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff92400e7e0 tx=0x7ff92400eaf0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 shutdown_connections 2023-11-24T06:20:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff90c06af60 0x7ff90c06d420 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281446a0 0x7ff9281f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff92806c540 0x7ff9280cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 --2- 172.21.15.89:0/754027237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9280adf00 0x7ff9281f3ce0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:44.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 >> 172.21.15.89:0/754027237 conn(0x7ff928071b90 msgr2=0x7ff92815e710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:44.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 shutdown_connections 2023-11-24T06:20:44.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:44.984+0000 7ff930306700 1 -- 172.21.15.89:0/754027237 wait complete. 2023-11-24T06:20:44.999 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T06:20:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.118+0000 7fcf9d87e700 1 Processor -- start 2023-11-24T06:20:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.118+0000 7fcf9d87e700 1 -- start start 2023-11-24T06:20:45.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.118+0000 7fcf9d87e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcf9815e480 0x7fcf980d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.118+0000 7fcf9d87e700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980d2a70 con 0x7fcf980d3610 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.118+0000 7fcf9d87e700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980d2d50 con 0x7fcf980b6060 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.118+0000 7fcf9d87e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf980d3030 con 0x7fcf9815e480 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.118+0000 7fcf967fc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcf980d3610 0x7fcf980b5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42068/0 (socket says 172.21.15.89:42068) 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.118+0000 7fcf967fc700 1 -- 172.21.15.89:0/2253161587 learned_addr learned my addr 172.21.15.89:0/2253161587 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf96ffd700 1 --2- 172.21.15.89:0/2253161587 >> v2:172.21.15.123:3300/0 conn(0x7fcf9815e480 0x7fcf980d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430366723 0 0) 0x7fcf980d2a70 con 0x7fcf980d3610 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf800035b0 con 0x7fcf980d3610 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 72779478 0 0) 0x7fcf800035b0 con 0x7fcf980d3610 2023-11-24T06:20:45.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 >> v2:172.21.15.123:3300/0 conn(0x7fcf9815e480 msgr2=0x7fcf980d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 --2- 172.21.15.89:0/2253161587 >> v2:172.21.15.123:3300/0 conn(0x7fcf9815e480 0x7fcf980d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 >> v1:172.21.15.123:6789/0 conn(0x7fcf980b6060 legacy=0x7fcf980b7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980b7340 con 0x7fcf980d3610 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fcf8c001670 con 0x7fcf980d3610 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 >> v1:172.21.15.89:6789/0 conn(0x7fcf980d3610 legacy=0x7fcf980b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf80003a00 0x7fcf80003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80004850 0x7fcf80004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80005210 0x7fcf8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980d3030 con 0x7fcf80004850 2023-11-24T06:20:45.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8000c1b0 con 0x7fcf80003a00 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8000c370 con 0x7fcf80005210 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf967fc700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80004850 0x7fcf80004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf96ffd700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf80003a00 0x7fcf80003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf977fe700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80005210 0x7fcf8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf967fc700 1 -- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80005210 msgr2=0x7fcf8000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf967fc700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80005210 0x7fcf8000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf967fc700 1 -- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf80003a00 msgr2=0x7fcf80003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf967fc700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf80003a00 0x7fcf80003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.119+0000 7fcf967fc700 1 -- 172.21.15.89:0/2253161587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800035b0 con 0x7fcf80004850 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf977fe700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80005210 0x7fcf8000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf977fe700 1 -- 172.21.15.89:0/2253161587 reap_dead start 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf967fc700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80004850 0x7fcf80004cd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00c240 tx=0x7fcf80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b3ecfc564821e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c007c10 con 0x7fcf80004850 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c00ce00 con 0x7fcf80004850 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf95ffb700 1 -- 172.21.15.89:0/2253161587 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c005ba0 con 0x7fcf80004850 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf9d87e700 1 -- 172.21.15.89:0/2253161587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80004850 msgr2=0x7fcf80004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf9d87e700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80004850 0x7fcf80004cd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00c240 tx=0x7fcf80005880 comp rx=0 tx=0).stop 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf9d87e700 1 -- 172.21.15.89:0/2253161587 shutdown_connections 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf9d87e700 1 --2- 172.21.15.89:0/2253161587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf80004850 0x7fcf80004cd0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf9d87e700 1 -- 172.21.15.89:0/2253161587 >> 172.21.15.89:0/2253161587 conn(0x7fcf980d0950 msgr2=0x7fcf980d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf9d87e700 1 -- 172.21.15.89:0/2253161587 shutdown_connections 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.120+0000 7fcf9d87e700 1 -- 172.21.15.89:0/2253161587 wait complete. 2023-11-24T06:20:45.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.121+0000 7fcf9d87e700 1 Processor -- start 2023-11-24T06:20:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.122+0000 7fcf9d87e700 1 -- start start 2023-11-24T06:20:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.122+0000 7fcf9d87e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80004850 0x7fcf98151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.122+0000 7fcf9d87e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf980c8610 0x7fcf980c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.122+0000 7fcf9d87e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf980cd590 0x7fcf980c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.122+0000 7fcf9d87e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98151db0 con 0x7fcf980cd590 2023-11-24T06:20:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.122+0000 7fcf9d87e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980c8c20 con 0x7fcf980c8610 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.122+0000 7fcf9d87e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980cafa0 con 0x7fcf80004850 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf980cd590 0x7fcf980c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf980cd590 0x7fcf980c7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38576/0 (socket says 172.21.15.89:38576) 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 -- 172.21.15.89:0/1317202348 learned_addr learned my addr 172.21.15.89:0/1317202348 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf967fc700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf980c8610 0x7fcf980c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf96ffd700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80004850 0x7fcf98151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 -- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80004850 msgr2=0x7fcf98151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80004850 0x7fcf98151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 -- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf980c8610 msgr2=0x7fcf980c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf980c8610 0x7fcf980c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 -- 172.21.15.89:0/1317202348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980c8ec0 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf96ffd700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80004850 0x7fcf98151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf977fe700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf980cd590 0x7fcf980c7f40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcf980d1df0 tx=0x7fcf9000ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf7ffff700 1 -- 172.21.15.89:0/1317202348 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90003e10 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf7ffff700 1 -- 172.21.15.89:0/1317202348 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf900020e0 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf981606f0 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf7ffff700 1 -- 172.21.15.89:0/1317202348 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900052b0 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.123+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf98160cb0 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.124+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf980751e0 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.125+0000 7fcf7ffff700 1 -- 172.21.15.89:0/1317202348 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fcf90005410 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.125+0000 7fcf7ffff700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcf8406af60 0x7fcf8406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.125+0000 7fcf7ffff700 1 -- 172.21.15.89:0/1317202348 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf90089160 con 0x7fcf980cd590 2023-11-24T06:20:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.125+0000 7fcf96ffd700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcf8406af60 0x7fcf8406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.125+0000 7fcf96ffd700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcf8406af60 0x7fcf8406d420 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009ce0 tx=0x7fcf88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.129+0000 7fcf7ffff700 1 -- 172.21.15.89:0/1317202348 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf90054110 con 0x7fcf980cd590 2023-11-24T06:20:45.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.365+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcf980b4090 con 0x7fcf980cd590 2023-11-24T06:20:45.368 INFO:tasks.ceph.mon.a.smithi089.stderr:2023-11-24T06:20:45.366+0000 7f7eccc4c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T06:20:45.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.369+0000 7fcf7ffff700 1 -- 172.21.15.89:0/1317202348 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcf90057730 con 0x7fcf980cd590 2023-11-24T06:20:45.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcf8406af60 msgr2=0x7fcf8406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcf8406af60 0x7fcf8406d420 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009ce0 tx=0x7fcf88008040 comp rx=0 tx=0).stop 2023-11-24T06:20:45.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf980cd590 msgr2=0x7fcf980c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf980cd590 0x7fcf980c7f40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcf980d1df0 tx=0x7fcf9000ddc0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 shutdown_connections 2023-11-24T06:20:45.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf980cd590 0x7fcf980c7f40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf80004850 0x7fcf98151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fcf8406af60 0x7fcf8406d420 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 --2- 172.21.15.89:0/1317202348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf980c8610 0x7fcf980c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 >> 172.21.15.89:0/1317202348 conn(0x7fcf980d0950 msgr2=0x7fcf980b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:45.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 shutdown_connections 2023-11-24T06:20:45.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.371+0000 7fcf9d87e700 1 -- 172.21.15.89:0/1317202348 wait complete. 2023-11-24T06:20:45.375 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T06:20:45.377 INFO:tasks.ceph.mds.c.smithi089.stderr: -4585> 2023-11-24T06:20:28.608+0000 7f681759b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:20:45.388 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T06:20:45.406 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-11-24T06:20:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.507+0000 7f637817a700 1 Processor -- start 2023-11-24T06:20:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.507+0000 7f637817a700 1 -- start start 2023-11-24T06:20:45.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.508+0000 7f637817a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700b4020 0x7f6370073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.508+0000 7f637817a700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637015bf70 con 0x7f63700af520 2023-11-24T06:20:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.508+0000 7f637817a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637015ee00 con 0x7f63700744d0 2023-11-24T06:20:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.508+0000 7f637817a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6370074980 con 0x7f63700b4020 2023-11-24T06:20:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.508+0000 7f6375f16700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f63700af520 0x7f63700afb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42098/0 (socket says 172.21.15.89:42098) 2023-11-24T06:20:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.508+0000 7f6375f16700 1 -- 172.21.15.89:0/2319658207 learned_addr learned my addr 172.21.15.89:0/2319658207 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6375715700 1 --2- 172.21.15.89:0/2319658207 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700b4020 0x7f6370073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6374f14700 1 -- 172.21.15.89:0/2319658207 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074205372 0 0) 0x7f6360001110 con 0x7f63700af520 2023-11-24T06:20:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6374f14700 1 -- 172.21.15.89:0/2319658207 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f635c0035b0 con 0x7f63700af520 2023-11-24T06:20:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6374f14700 1 -- 172.21.15.89:0/2319658207 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439044478 0 0) 0x7f637015ee00 con 0x7f63700744d0 2023-11-24T06:20:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6374f14700 1 -- 172.21.15.89:0/2319658207 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6360001110 con 0x7f63700744d0 2023-11-24T06:20:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6374f14700 1 -- 172.21.15.89:0/2319658207 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991198014 0 0) 0x7f635c0035b0 con 0x7f63700af520 2023-11-24T06:20:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6375715700 1 -- 172.21.15.89:0/2319658207 >> v1:172.21.15.123:6789/0 conn(0x7f63700744d0 legacy=0x7f637006d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6375715700 1 -- 172.21.15.89:0/2319658207 >> v1:172.21.15.89:6789/0 conn(0x7f63700af520 legacy=0x7f63700afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6375715700 1 -- 172.21.15.89:0/2319658207 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63700b0240 con 0x7f63700b4020 2023-11-24T06:20:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6375715700 1 --2- 172.21.15.89:0/2319658207 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700b4020 0x7f6370073f90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f636c009870 tx=0x7f636c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a4e4822a9bb15 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6374f14700 1 -- 172.21.15.89:0/2319658207 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6360001800 con 0x7f63700b4020 2023-11-24T06:20:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6374f14700 1 -- 172.21.15.89:0/2319658207 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c004030 con 0x7f63700b4020 2023-11-24T06:20:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.509+0000 7f6374f14700 1 -- 172.21.15.89:0/2319658207 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c0043a0 con 0x7f63700b4020 2023-11-24T06:20:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.510+0000 7f637817a700 1 -- 172.21.15.89:0/2319658207 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700b4020 msgr2=0x7f6370073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.510+0000 7f637817a700 1 --2- 172.21.15.89:0/2319658207 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700b4020 0x7f6370073f90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f636c009870 tx=0x7f636c009b80 comp rx=0 tx=0).stop 2023-11-24T06:20:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.510+0000 7f637817a700 1 -- 172.21.15.89:0/2319658207 shutdown_connections 2023-11-24T06:20:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.510+0000 7f637817a700 1 --2- 172.21.15.89:0/2319658207 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700b4020 0x7f6370073f90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.510+0000 7f637817a700 1 -- 172.21.15.89:0/2319658207 >> 172.21.15.89:0/2319658207 conn(0x7f63700b28f0 msgr2=0x7f63700b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.510+0000 7f637817a700 1 -- 172.21.15.89:0/2319658207 shutdown_connections 2023-11-24T06:20:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.510+0000 7f637817a700 1 -- 172.21.15.89:0/2319658207 wait complete. 2023-11-24T06:20:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.511+0000 7f637817a700 1 Processor -- start 2023-11-24T06:20:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.511+0000 7f637817a700 1 -- start start 2023-11-24T06:20:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f637817a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700744d0 0x7f637015f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f637817a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63700af520 0x7f6370150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f637817a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63700b4020 0x7f6370154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f637817a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6370156310 con 0x7f63700b4020 2023-11-24T06:20:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f637817a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6370157310 con 0x7f63700af520 2023-11-24T06:20:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f637817a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6370155a70 con 0x7f63700744d0 2023-11-24T06:20:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63700b4020 0x7f6370154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63700b4020 0x7f6370154ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38604/0 (socket says 172.21.15.89:38604) 2023-11-24T06:20:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 -- 172.21.15.89:0/3373545504 learned_addr learned my addr 172.21.15.89:0/3373545504 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6375715700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63700af520 0x7f6370150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6375f16700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700744d0 0x7f637015f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 -- 172.21.15.89:0/3373545504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700744d0 msgr2=0x7f637015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700744d0 0x7f637015f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 -- 172.21.15.89:0/3373545504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63700af520 msgr2=0x7f6370150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63700af520 0x7f6370150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 -- 172.21.15.89:0/3373545504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0094a0 con 0x7f63700b4020 2023-11-24T06:20:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.512+0000 7f6376717700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63700b4020 0x7f6370154ef0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f636800cc20 tx=0x7f636800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.513+0000 7f63667fc700 1 -- 172.21.15.89:0/3373545504 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636800ac20 con 0x7f63700b4020 2023-11-24T06:20:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.513+0000 7f63667fc700 1 -- 172.21.15.89:0/3373545504 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636800ad80 con 0x7f63700b4020 2023-11-24T06:20:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.513+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6370155d50 con 0x7f63700b4020 2023-11-24T06:20:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.513+0000 7f63667fc700 1 -- 172.21.15.89:0/3373545504 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6368003c20 con 0x7f63700b4020 2023-11-24T06:20:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.513+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637015d260 con 0x7f63700b4020 2023-11-24T06:20:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.513+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6370068480 con 0x7f63700b4020 2023-11-24T06:20:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.514+0000 7f63667fc700 1 -- 172.21.15.89:0/3373545504 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f63680050d0 con 0x7f63700b4020 2023-11-24T06:20:45.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.514+0000 7f63667fc700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f635406af60 0x7f635406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:45.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.514+0000 7f63667fc700 1 -- 172.21.15.89:0/3373545504 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6368089740 con 0x7f63700b4020 2023-11-24T06:20:45.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.514+0000 7f6375f16700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f635406af60 0x7f635406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:45.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.515+0000 7f6375f16700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f635406af60 0x7f635406d420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6360001930 tx=0x7f6360001d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.519+0000 7f63667fc700 1 -- 172.21.15.89:0/3373545504 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6368054830 con 0x7f63700b4020 2023-11-24T06:20:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:45.752+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6370157480 con 0x7f63700b4020 2023-11-24T06:20:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.375+0000 7f63667fc700 1 -- 172.21.15.89:0/3373545504 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6368057e50 con 0x7f63700b4020 2023-11-24T06:20:46.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f635406af60 msgr2=0x7f635406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:46.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f635406af60 0x7f635406d420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6360001930 tx=0x7f6360001d30 comp rx=0 tx=0).stop 2023-11-24T06:20:46.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63700b4020 msgr2=0x7f6370154ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:46.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63700b4020 0x7f6370154ef0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f636800cc20 tx=0x7f636800a060 comp rx=0 tx=0).stop 2023-11-24T06:20:46.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 shutdown_connections 2023-11-24T06:20:46.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63700b4020 0x7f6370154ef0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63700744d0 0x7f637015f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f635406af60 0x7f635406d420 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 --2- 172.21.15.89:0/3373545504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63700af520 0x7f6370150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 >> 172.21.15.89:0/3373545504 conn(0x7f63700b28f0 msgr2=0x7f63700b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:46.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 shutdown_connections 2023-11-24T06:20:46.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.378+0000 7f637817a700 1 -- 172.21.15.89:0/3373545504 wait complete. 2023-11-24T06:20:46.395 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T06:20:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.558+0000 7f10a63c1700 1 Processor -- start 2023-11-24T06:20:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.558+0000 7f10a63c1700 1 -- start start 2023-11-24T06:20:46.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f10a63c1700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f10a0145620 0x7f10a006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:46.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f10a63c1700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a00cf530 con 0x7f10a0073f90 2023-11-24T06:20:46.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f10a63c1700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a006b8f0 con 0x7f10a00cd810 2023-11-24T06:20:46.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f10a63c1700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a006b700 con 0x7f10a0145620 2023-11-24T06:20:46.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f10a495e700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f10a0073f90 0x7f10a0074390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42118/0 (socket says 172.21.15.89:42118) 2023-11-24T06:20:46.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f10a495e700 1 -- 172.21.15.89:0/804105599 learned_addr learned my addr 172.21.15.89:0/804105599 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:46.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f109ffff700 1 --2- 172.21.15.89:0/804105599 >> v2:172.21.15.123:3300/0 conn(0x7f10a0145620 0x7f10a006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964411332 0 0) 0x7f10a00cf530 con 0x7f10a0073f90 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10880035b0 con 0x7f10a0073f90 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.559+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2719128157 0 0) 0x7f10880035b0 con 0x7f10a0073f90 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 >> v2:172.21.15.123:3300/0 conn(0x7f10a0145620 msgr2=0x7f10a006dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 --2- 172.21.15.89:0/804105599 >> v2:172.21.15.123:3300/0 conn(0x7f10a0145620 0x7f10a006dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 >> v1:172.21.15.123:6789/0 conn(0x7f10a00cd810 legacy=0x7f10a006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a006bbd0 con 0x7f10a0073f90 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f10980019f0 con 0x7f10a0073f90 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 >> v1:172.21.15.89:6789/0 conn(0x7f10a0073f90 legacy=0x7f10a0074390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1088003a00 0x7f1088003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088004850 0x7f1088004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1088005210 0x7f108800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108800c080 con 0x7f1088003a00 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108800c1f0 con 0x7f1088004850 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108800c3b0 con 0x7f1088005210 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109ffff700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1088003a00 0x7f1088003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109f7fe700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088004850 0x7f1088004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f10a495e700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1088005210 0x7f108800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109ffff700 1 -- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1088005210 msgr2=0x7f108800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109ffff700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1088005210 0x7f108800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109ffff700 1 -- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088004850 msgr2=0x7f1088004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109ffff700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088004850 0x7f1088004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109ffff700 1 -- 172.21.15.89:0/804105599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880035b0 con 0x7f1088003a00 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f10a495e700 1 -- 172.21.15.89:0/804105599 reap_dead start 2023-11-24T06:20:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109f7fe700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088004850 0x7f1088004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109ffff700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1088003a00 0x7f1088003e20 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1094009b60 tx=0x7f1088005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd02eaee614e212 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109400e6c0 con 0x7f1088003a00 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109400e820 con 0x7f1088003a00 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.560+0000 7f109effd700 1 -- 172.21.15.89:0/804105599 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109400eb90 con 0x7f1088003a00 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.561+0000 7f10a63c1700 1 -- 172.21.15.89:0/804105599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1088003a00 msgr2=0x7f1088003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.561+0000 7f10a63c1700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1088003a00 0x7f1088003e20 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1094009b60 tx=0x7f1088005880 comp rx=0 tx=0).stop 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.561+0000 7f10a63c1700 1 -- 172.21.15.89:0/804105599 shutdown_connections 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.561+0000 7f10a63c1700 1 --2- 172.21.15.89:0/804105599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1088003a00 0x7f1088003e20 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.561+0000 7f10a63c1700 1 -- 172.21.15.89:0/804105599 >> 172.21.15.89:0/804105599 conn(0x7f10a00d0b40 msgr2=0x7f10a00d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.561+0000 7f10a63c1700 1 -- 172.21.15.89:0/804105599 shutdown_connections 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.561+0000 7f10a63c1700 1 -- 172.21.15.89:0/804105599 wait complete. 2023-11-24T06:20:46.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.562+0000 7f10a63c1700 1 Processor -- start 2023-11-24T06:20:46.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.562+0000 7f10a63c1700 1 -- start start 2023-11-24T06:20:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a63c1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088003a00 0x7f10a00cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a63c1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10a015e6d0 0x7f10a00b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a63c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a01f7350 0x7f10a01f7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a63c1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00b3c50 con 0x7f10a01f7350 2023-11-24T06:20:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a63c1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00cbe20 con 0x7f1088003a00 2023-11-24T06:20:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a63c1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00cc100 con 0x7f10a015e6d0 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a01f7350 0x7f10a01f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a01f7350 0x7f10a01f7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38626/0 (socket says 172.21.15.89:38626) 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 -- 172.21.15.89:0/1358821746 learned_addr learned my addr 172.21.15.89:0/1358821746 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f109f7fe700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10a015e6d0 0x7f10a00b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f109ffff700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088003a00 0x7f10a00cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 -- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10a015e6d0 msgr2=0x7f10a00b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10a015e6d0 0x7f10a00b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 -- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088003a00 msgr2=0x7f10a00cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088003a00 0x7f10a00cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 -- 172.21.15.89:0/1358821746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a00cc3e0 con 0x7f10a01f7350 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f109ffff700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088003a00 0x7f10a00cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.563+0000 7f10a495e700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a01f7350 0x7f10a01f7800 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f10a00d1730 tx=0x7f109800e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.564+0000 7f109cff9700 1 -- 172.21.15.89:0/1358821746 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109800eb30 con 0x7f10a01f7350 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.564+0000 7f109cff9700 1 -- 172.21.15.89:0/1358821746 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109800ec90 con 0x7f10a01f7350 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.564+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a00cc6c0 con 0x7f10a01f7350 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.564+0000 7f109cff9700 1 -- 172.21.15.89:0/1358821746 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1098013420 con 0x7f10a01f7350 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.564+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a00b19f0 con 0x7f10a01f7350 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.565+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a00b1c10 con 0x7f10a01f7350 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.565+0000 7f109cff9700 1 -- 172.21.15.89:0/1358821746 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f1098013580 con 0x7f10a01f7350 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.565+0000 7f109cff9700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f108c06af60 0x7f108c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.565+0000 7f109cff9700 1 -- 172.21.15.89:0/1358821746 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1098089a70 con 0x7f10a01f7350 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.565+0000 7f109ffff700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f108c06af60 0x7f108c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.566+0000 7f109ffff700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f108c06af60 0x7f108c06d420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f10940090b0 tx=0x7f1094008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.570+0000 7f109cff9700 1 -- 172.21.15.89:0/1358821746 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10980585f0 con 0x7f10a01f7350 2023-11-24T06:20:46.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:46.765+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10a00b3d90 con 0x7f10a01f7350 2023-11-24T06:20:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.380+0000 7f109cff9700 1 -- 172.21.15.89:0/1358821746 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v55) v1 ==== 181+0+0 (secure 0 0 0) 0x7f10a00b3d90 con 0x7f10a01f7350 2023-11-24T06:20:47.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f108c06af60 msgr2=0x7f108c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:47.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f108c06af60 0x7f108c06d420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f10940090b0 tx=0x7f1094008030 comp rx=0 tx=0).stop 2023-11-24T06:20:47.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a01f7350 msgr2=0x7f10a01f7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:47.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a01f7350 0x7f10a01f7800 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f10a00d1730 tx=0x7f109800e440 comp rx=0 tx=0).stop 2023-11-24T06:20:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 shutdown_connections 2023-11-24T06:20:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a01f7350 0x7f10a01f7800 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f108c06af60 0x7f108c06d420 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1088003a00 0x7f10a00cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 --2- 172.21.15.89:0/1358821746 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10a015e6d0 0x7f10a00b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 >> 172.21.15.89:0/1358821746 conn(0x7f10a00d0b40 msgr2=0x7f10a006e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 shutdown_connections 2023-11-24T06:20:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.382+0000 7f10a63c1700 1 -- 172.21.15.89:0/1358821746 wait complete. 2023-11-24T06:20:47.387 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2023-11-24T06:20:47.402 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T06:20:47.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.525+0000 7fb9006af700 1 Processor -- start 2023-11-24T06:20:47.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.525+0000 7fb9006af700 1 -- start start 2023-11-24T06:20:47.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb9006af700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb8f80b01e0 0x7fb8f806e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:47.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb9006af700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f80b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:47.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb9006af700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8f8073f90 0x7fb8f80743e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:47.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb9006af700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f81470a0 con 0x7fb8f8073f90 2023-11-24T06:20:47.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb9006af700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f8074920 con 0x7fb8f80b01e0 2023-11-24T06:20:47.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb9006af700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f80b5e40 con 0x7fb8f80b59c0 2023-11-24T06:20:47.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8f8073f90 0x7fb8f80743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:47.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8f8073f90 0x7fb8f80743e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38634/0 (socket says 172.21.15.89:38634) 2023-11-24T06:20:47.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 -- 172.21.15.89:0/103196594 learned_addr learned my addr 172.21.15.89:0/103196594 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:47.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 -- 172.21.15.89:0/103196594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 msgr2=0x7fb8f80b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:47.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fe44b700 1 --2- 172.21.15.89:0/103196594 >> v2:172.21.15.123:3300/0 conn(0x7fb8f80b01e0 0x7fb8f806e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:47.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fdc4a700 1 --2- 172.21.15.89:0/103196594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f80b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:47.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 --2- 172.21.15.89:0/103196594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f80b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 -- 172.21.15.89:0/103196594 >> v2:172.21.15.123:3300/0 conn(0x7fb8f80b01e0 msgr2=0x7fb8f806e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:47.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 --2- 172.21.15.89:0/103196594 >> v2:172.21.15.123:3300/0 conn(0x7fb8f80b01e0 0x7fb8f806e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 -- 172.21.15.89:0/103196594 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f80b05c0 con 0x7fb8f8073f90 2023-11-24T06:20:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fe44b700 1 --2- 172.21.15.89:0/103196594 >> v2:172.21.15.123:3300/0 conn(0x7fb8f80b01e0 0x7fb8f806e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fdc4a700 1 --2- 172.21.15.89:0/103196594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f80b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.526+0000 7fb8fec4c700 1 --2- 172.21.15.89:0/103196594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f8073f90 0x7fb8f80743e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8072ce0 tx=0x7fb8f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2650a9795a0a9293 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb8fd449700 1 -- 172.21.15.89:0/103196594 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f000e070 con 0x7fb8f8073f90 2023-11-24T06:20:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb8fd449700 1 -- 172.21.15.89:0/103196594 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f000b9c0 con 0x7fb8f8073f90 2023-11-24T06:20:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb8fd449700 1 -- 172.21.15.89:0/103196594 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f000bd70 con 0x7fb8f8073f90 2023-11-24T06:20:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 -- 172.21.15.89:0/103196594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f8073f90 msgr2=0x7fb8f80743e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 --2- 172.21.15.89:0/103196594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f8073f90 0x7fb8f80743e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8072ce0 tx=0x7fb8f000b330 comp rx=0 tx=0).stop 2023-11-24T06:20:47.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 -- 172.21.15.89:0/103196594 shutdown_connections 2023-11-24T06:20:47.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 --2- 172.21.15.89:0/103196594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f8073f90 0x7fb8f80743e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 --2- 172.21.15.89:0/103196594 >> v2:172.21.15.123:3300/0 conn(0x7fb8f80b01e0 0x7fb8f806e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 --2- 172.21.15.89:0/103196594 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f80b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 -- 172.21.15.89:0/103196594 >> 172.21.15.89:0/103196594 conn(0x7fb8f80d0b40 msgr2=0x7fb8f80d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:47.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 -- 172.21.15.89:0/103196594 shutdown_connections 2023-11-24T06:20:47.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.527+0000 7fb9006af700 1 -- 172.21.15.89:0/103196594 wait complete. 2023-11-24T06:20:47.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.528+0000 7fb9006af700 1 Processor -- start 2023-11-24T06:20:47.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.528+0000 7fb9006af700 1 -- start start 2023-11-24T06:20:47.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb9006af700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8f8073f90 0x7fb8f81f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:47.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb9006af700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f80b01e0 0x7fb8f815fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:47.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb9006af700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f8134ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:47.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb9006af700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f81367e0 con 0x7fb8f80b01e0 2023-11-24T06:20:47.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb9006af700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f8135b60 con 0x7fb8f8073f90 2023-11-24T06:20:47.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb9006af700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f8135e40 con 0x7fb8f80b59c0 2023-11-24T06:20:47.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fdc4a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f80b01e0 0x7fb8f815fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:47.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fdc4a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f80b01e0 0x7fb8f815fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38638/0 (socket says 172.21.15.89:38638) 2023-11-24T06:20:47.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fdc4a700 1 -- 172.21.15.89:0/4038930182 learned_addr learned my addr 172.21.15.89:0/4038930182 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:47.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fe44b700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8f8073f90 0x7fb8f81f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:47.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fec4c700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f8134ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fdc4a700 1 -- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 msgr2=0x7fb8f8134ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fdc4a700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f8134ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fdc4a700 1 -- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8f8073f90 msgr2=0x7fb8f81f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fdc4a700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8f8073f90 0x7fb8f81f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fdc4a700 1 -- 172.21.15.89:0/4038930182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0009b30 con 0x7fb8f80b01e0 2023-11-24T06:20:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.529+0000 7fb8fec4c700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f8134ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.530+0000 7fb8fdc4a700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f80b01e0 0x7fb8f815fbf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb8f400c750 tx=0x7fb8f400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.530+0000 7fb8eeffd700 1 -- 172.21.15.89:0/4038930182 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4011800 con 0x7fb8f80b01e0 2023-11-24T06:20:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.530+0000 7fb8eeffd700 1 -- 172.21.15.89:0/4038930182 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f4011960 con 0x7fb8f80b01e0 2023-11-24T06:20:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.530+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f8136120 con 0x7fb8f80b01e0 2023-11-24T06:20:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.530+0000 7fb8eeffd700 1 -- 172.21.15.89:0/4038930182 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4011cd0 con 0x7fb8f80b01e0 2023-11-24T06:20:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.530+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f8137e60 con 0x7fb8f80b01e0 2023-11-24T06:20:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.531+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f8067f60 con 0x7fb8f80b01e0 2023-11-24T06:20:47.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.531+0000 7fb8eeffd700 1 -- 172.21.15.89:0/4038930182 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fb8f4021350 con 0x7fb8f80b01e0 2023-11-24T06:20:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.532+0000 7fb8eeffd700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8dc06af60 0x7fb8dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.532+0000 7fb8eeffd700 1 -- 172.21.15.89:0/4038930182 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fb8f40894a0 con 0x7fb8f80b01e0 2023-11-24T06:20:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.532+0000 7fb8fe44b700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8dc06af60 0x7fb8dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.532+0000 7fb8fe44b700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8dc06af60 0x7fb8dc06d420 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8006fd0 tx=0x7fb8e8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.536+0000 7fb8eeffd700 1 -- 172.21.15.89:0/4038930182 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8f4054640 con 0x7fb8f80b01e0 2023-11-24T06:20:47.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:47.727+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb8f8136920 con 0x7fb8f80b01e0 2023-11-24T06:20:48.054 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:20:48.056 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:20:48.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.385+0000 7fb8eeffd700 1 -- 172.21.15.89:0/4038930182 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v56) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb8f8136920 con 0x7fb8f80b01e0 2023-11-24T06:20:48.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8dc06af60 msgr2=0x7fb8dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8dc06af60 0x7fb8dc06d420 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8006fd0 tx=0x7fb8e8008040 comp rx=0 tx=0).stop 2023-11-24T06:20:48.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f80b01e0 msgr2=0x7fb8f815fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f80b01e0 0x7fb8f815fbf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb8f400c750 tx=0x7fb8f400cb10 comp rx=0 tx=0).stop 2023-11-24T06:20:48.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 shutdown_connections 2023-11-24T06:20:48.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8f80b59c0 0x7fb8f8134ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb8dc06af60 0x7fb8dc06d420 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8f8073f90 0x7fb8f81f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 --2- 172.21.15.89:0/4038930182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8f80b01e0 0x7fb8f815fbf0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 >> 172.21.15.89:0/4038930182 conn(0x7fb8f80d0b40 msgr2=0x7fb8f80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:48.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 shutdown_connections 2023-11-24T06:20:48.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.388+0000 7fb9006af700 1 -- 172.21.15.89:0/4038930182 wait complete. 2023-11-24T06:20:48.393 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2023-11-24T06:20:48.407 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T06:20:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.528+0000 7f0a0b018700 1 Processor -- start 2023-11-24T06:20:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.528+0000 7f0a0b018700 1 -- start start 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a0b018700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0a0406ecf0 0x7f0a04165180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a0b018700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040b2f20 con 0x7f0a040cc540 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a0b018700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0415aa00 con 0x7f0a0415a620 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a0b018700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a040ce960 con 0x7f0a0406ecf0 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a03fff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0a040cc540 0x7f0a0406e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42148/0 (socket says 172.21.15.89:42148) 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a03fff700 1 -- 172.21.15.89:0/2197282307 learned_addr learned my addr 172.21.15.89:0/2197282307 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a095b5700 1 --2- 172.21.15.89:0/2197282307 >> v2:172.21.15.123:3300/0 conn(0x7f0a0406ecf0 0x7f0a04165180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143279644 0 0) 0x7f0a040b2f20 con 0x7f0a040cc540 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f40035b0 con 0x7f0a040cc540 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.529+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2582454652 0 0) 0x7f09f40035b0 con 0x7f0a040cc540 2023-11-24T06:20:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 >> v2:172.21.15.123:3300/0 conn(0x7f0a0406ecf0 msgr2=0x7f0a04165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 --2- 172.21.15.89:0/2197282307 >> v2:172.21.15.123:3300/0 conn(0x7f0a0406ecf0 0x7f0a04165180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 >> v1:172.21.15.123:6789/0 conn(0x7f0a0415a620 legacy=0x7f0a040ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a040cec40 con 0x7f0a040cc540 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f09f0001670 con 0x7f0a040cc540 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 >> v1:172.21.15.89:6789/0 conn(0x7f0a040cc540 legacy=0x7f0a0406e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09f4003a00 0x7f09f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09f4004850 0x7f09f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4005210 0x7f09f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a040ce960 con 0x7f09f4004850 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f400c0b0 con 0x7f09f4005210 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f400c270 con 0x7f09f4003a00 2023-11-24T06:20:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a03fff700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09f4004850 0x7f09f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a095b5700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4005210 0x7f09f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a08db4700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09f4003a00 0x7f09f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a03fff700 1 -- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09f4003a00 msgr2=0x7f09f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a03fff700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09f4003a00 0x7f09f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a03fff700 1 -- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4005210 msgr2=0x7f09f400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a03fff700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4005210 0x7f09f400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a03fff700 1 -- 172.21.15.89:0/2197282307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f40035b0 con 0x7f09f4004850 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a095b5700 1 -- 172.21.15.89:0/2197282307 reap_dead start 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a08db4700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09f4003a00 0x7f09f4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a03fff700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09f4004850 0x7f09f4004cd0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f09f000c240 tx=0x7f09f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ce88a75fa3fb85 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0007c10 con 0x7f09f4004850 2023-11-24T06:20:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f000cd60 con 0x7f09f4004850 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.530+0000 7f0a037fe700 1 -- 172.21.15.89:0/2197282307 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0005ba0 con 0x7f09f4004850 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.531+0000 7f0a0b018700 1 -- 172.21.15.89:0/2197282307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09f4004850 msgr2=0x7f09f4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.531+0000 7f0a0b018700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09f4004850 0x7f09f4004cd0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f09f000c240 tx=0x7f09f4005880 comp rx=0 tx=0).stop 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.531+0000 7f0a0b018700 1 -- 172.21.15.89:0/2197282307 shutdown_connections 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.531+0000 7f0a0b018700 1 --2- 172.21.15.89:0/2197282307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09f4004850 0x7f09f4004cd0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.531+0000 7f0a0b018700 1 -- 172.21.15.89:0/2197282307 >> 172.21.15.89:0/2197282307 conn(0x7f0a040b2050 msgr2=0x7f0a040b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.531+0000 7f0a0b018700 1 -- 172.21.15.89:0/2197282307 shutdown_connections 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.531+0000 7f0a0b018700 1 -- 172.21.15.89:0/2197282307 wait complete. 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.532+0000 7f0a0b018700 1 Processor -- start 2023-11-24T06:20:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.532+0000 7f0a0b018700 1 -- start start 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a0b018700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4004850 0x7f0a0414e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a0b018700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a0414b820 0x7f0a04136270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a0b018700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a040ac800 0x7f0a040aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a0b018700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a040ac2c0 con 0x7f0a040ac800 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a0b018700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a040ae890 con 0x7f09f4004850 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a0b018700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a041367b0 con 0x7f0a0414b820 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a095b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a040ac800 0x7f0a040aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a095b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a040ac800 0x7f0a040aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38656/0 (socket says 172.21.15.89:38656) 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a095b5700 1 -- 172.21.15.89:0/3834669711 learned_addr learned my addr 172.21.15.89:0/3834669711 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a03fff700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a0414b820 0x7f0a04136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a08db4700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4004850 0x7f0a0414e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a095b5700 1 -- 172.21.15.89:0/3834669711 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a0414b820 msgr2=0x7f0a04136270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a095b5700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a0414b820 0x7f0a04136270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a095b5700 1 -- 172.21.15.89:0/3834669711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4004850 msgr2=0x7f0a0414e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a095b5700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4004850 0x7f0a0414e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.533+0000 7f0a095b5700 1 -- 172.21.15.89:0/3834669711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a040ae580 con 0x7f0a040ac800 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.534+0000 7f0a095b5700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a040ac800 0x7f0a040aed20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0a040cfa00 tx=0x7f09fc011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.534+0000 7f0a017fa700 1 -- 172.21.15.89:0/3834669711 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc00e040 con 0x7f0a040ac800 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.534+0000 7f0a017fa700 1 -- 172.21.15.89:0/3834669711 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc00f040 con 0x7f0a040ac800 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.534+0000 7f0a017fa700 1 -- 172.21.15.89:0/3834669711 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc00fd30 con 0x7f0a040ac800 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.534+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a041369e0 con 0x7f0a040ac800 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.534+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a04136fa0 con 0x7f0a040ac800 2023-11-24T06:20:48.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.535+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a04068d90 con 0x7f0a040ac800 2023-11-24T06:20:48.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.535+0000 7f0a017fa700 1 -- 172.21.15.89:0/3834669711 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f09fc015430 con 0x7f0a040ac800 2023-11-24T06:20:48.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.535+0000 7f0a017fa700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09e806af60 0x7f09e806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.536+0000 7f0a017fa700 1 -- 172.21.15.89:0/3834669711 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09fc0899f0 con 0x7f0a040ac800 2023-11-24T06:20:48.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.536+0000 7f0a08db4700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09e806af60 0x7f09e806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.536+0000 7f0a08db4700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09e806af60 0x7f09e806d420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f09f8000c00 tx=0x7f09f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:48.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.540+0000 7f0a017fa700 1 -- 172.21.15.89:0/3834669711 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09fc08be60 con 0x7f0a040ac800 2023-11-24T06:20:48.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.731+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0a04068fb0 con 0x7f0a040ac800 2023-11-24T06:20:48.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.734+0000 7f0a017fa700 1 -- 172.21.15.89:0/3834669711 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f09fc0587e0 con 0x7f0a040ac800 2023-11-24T06:20:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09e806af60 msgr2=0x7f09e806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09e806af60 0x7f09e806d420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f09f8000c00 tx=0x7f09f8009040 comp rx=0 tx=0).stop 2023-11-24T06:20:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a040ac800 msgr2=0x7f0a040aed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a040ac800 0x7f0a040aed20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0a040cfa00 tx=0x7f09fc011ed0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 shutdown_connections 2023-11-24T06:20:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a040ac800 0x7f0a040aed20 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f09e806af60 0x7f09e806d420 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f4004850 0x7f0a0414e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 --2- 172.21.15.89:0/3834669711 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a0414b820 0x7f0a04136270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 >> 172.21.15.89:0/3834669711 conn(0x7f0a040b2050 msgr2=0x7f0a040b4000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 shutdown_connections 2023-11-24T06:20:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.736+0000 7f0a0b018700 1 -- 172.21.15.89:0/3834669711 wait complete. 2023-11-24T06:20:48.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_info' 2023-11-24T06:20:48.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.867+0000 7f759b776700 1 Processor -- start 2023-11-24T06:20:48.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.868+0000 7f759b776700 1 -- start start 2023-11-24T06:20:48.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.868+0000 7f759b776700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f759415bef0 0x7f759415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.868+0000 7f759b776700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759406dd80 con 0x7f75941624d0 2023-11-24T06:20:48.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.868+0000 7f759b776700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759415f880 con 0x7f75940afe70 2023-11-24T06:20:48.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.868+0000 7f759b776700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759415ed90 con 0x7f759415bef0 2023-11-24T06:20:48.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.868+0000 7f7599d13700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f75941624d0 0x7f759406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42164/0 (socket says 172.21.15.89:42164) 2023-11-24T06:20:48.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599d13700 1 -- 172.21.15.89:0/1638965922 learned_addr learned my addr 172.21.15.89:0/1638965922 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:48.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599512700 1 --2- 172.21.15.89:0/1638965922 >> v2:172.21.15.123:3300/0 conn(0x7f759415bef0 0x7f759415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864834201 0 0) 0x7f759406dd80 con 0x7f75941624d0 2023-11-24T06:20:48.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75800035b0 con 0x7f75941624d0 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47225957 0 0) 0x7f75800035b0 con 0x7f75941624d0 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 >> v2:172.21.15.123:3300/0 conn(0x7f759415bef0 msgr2=0x7f759415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 --2- 172.21.15.89:0/1638965922 >> v2:172.21.15.123:3300/0 conn(0x7f759415bef0 0x7f759415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 >> v1:172.21.15.123:6789/0 conn(0x7f75940afe70 legacy=0x7f75940b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759415c2d0 con 0x7f75941624d0 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f758c0019f0 con 0x7f75941624d0 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 >> v1:172.21.15.89:6789/0 conn(0x7f75941624d0 legacy=0x7f759406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7580003a00 0x7f7580003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580004400 0x7f75800048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7580005170 0x7f758000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759415ed90 con 0x7f7580003a00 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758000c0b0 con 0x7f7580005170 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758000c270 con 0x7f7580004400 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599512700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7580003a00 0x7f7580003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599d13700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7580005170 0x7f758000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7598d11700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580004400 0x7f75800048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599512700 1 -- 172.21.15.89:0/1638965922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580004400 msgr2=0x7f75800048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599512700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580004400 0x7f75800048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599512700 1 -- 172.21.15.89:0/1638965922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7580005170 msgr2=0x7f758000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599512700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7580005170 0x7f758000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599512700 1 -- 172.21.15.89:0/1638965922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800035b0 con 0x7f7580003a00 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.869+0000 7f7599d13700 1 -- 172.21.15.89:0/1638965922 reap_dead start 2023-11-24T06:20:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f7599512700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7580003a00 0x7f7580003de0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7584009b60 tx=0x7f7580005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef617d88956eeaa3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758400e6c0 con 0x7f7580003a00 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758400e820 con 0x7f7580003a00 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f758bfff700 1 -- 172.21.15.89:0/1638965922 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758400eb90 con 0x7f7580003a00 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f759b776700 1 -- 172.21.15.89:0/1638965922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7580003a00 msgr2=0x7f7580003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f759b776700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7580003a00 0x7f7580003de0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7584009b60 tx=0x7f7580005880 comp rx=0 tx=0).stop 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f759b776700 1 -- 172.21.15.89:0/1638965922 shutdown_connections 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f759b776700 1 --2- 172.21.15.89:0/1638965922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7580003a00 0x7f7580003de0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f759b776700 1 -- 172.21.15.89:0/1638965922 >> 172.21.15.89:0/1638965922 conn(0x7f7594071ac0 msgr2=0x7f7594071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f759b776700 1 -- 172.21.15.89:0/1638965922 shutdown_connections 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.870+0000 7f759b776700 1 -- 172.21.15.89:0/1638965922 wait complete. 2023-11-24T06:20:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f759b776700 1 Processor -- start 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f759b776700 1 -- start start 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f759b776700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580003a00 0x7f75940cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f759b776700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7594160ec0 0x7f7594161340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f759b776700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75940cdaa0 0x7f75940cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f759b776700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759414a820 con 0x7f7594160ec0 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f759b776700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594161880 con 0x7f75940cdaa0 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f759b776700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75940cf7e0 con 0x7f7580003a00 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f7598d11700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7594160ec0 0x7f7594161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f7598d11700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7594160ec0 0x7f7594161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38672/0 (socket says 172.21.15.89:38672) 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f7598d11700 1 -- 172.21.15.89:0/103678482 learned_addr learned my addr 172.21.15.89:0/103678482 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.872+0000 7f7599d13700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75940cdaa0 0x7f75940cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7599512700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580003a00 0x7f75940cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7598d11700 1 -- 172.21.15.89:0/103678482 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580003a00 msgr2=0x7f75940cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7598d11700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580003a00 0x7f75940cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7598d11700 1 -- 172.21.15.89:0/103678482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75940cdaa0 msgr2=0x7f75940cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7598d11700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75940cdaa0 0x7f75940cdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7598d11700 1 -- 172.21.15.89:0/103678482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940cfac0 con 0x7f7594160ec0 2023-11-24T06:20:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7598d11700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7594160ec0 0x7f7594161340 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f759000e800 tx=0x7f759000eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7589ffb700 1 -- 172.21.15.89:0/103678482 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75900052e0 con 0x7f7594160ec0 2023-11-24T06:20:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7589ffb700 1 -- 172.21.15.89:0/103678482 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590005440 con 0x7f7594160ec0 2023-11-24T06:20:48.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f7589ffb700 1 -- 172.21.15.89:0/103678482 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75900038c0 con 0x7f7594160ec0 2023-11-24T06:20:48.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75940cfda0 con 0x7f7594160ec0 2023-11-24T06:20:48.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.873+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75941e7690 con 0x7f7594160ec0 2023-11-24T06:20:48.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.874+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759406f910 con 0x7f7594160ec0 2023-11-24T06:20:48.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.874+0000 7f7589ffb700 1 -- 172.21.15.89:0/103678482 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f7590003a20 con 0x7f7594160ec0 2023-11-24T06:20:48.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.875+0000 7f7589ffb700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f757806af60 0x7f757806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:48.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.875+0000 7f7589ffb700 1 -- 172.21.15.89:0/103678482 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f75900889f0 con 0x7f7594160ec0 2023-11-24T06:20:48.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.875+0000 7f7599512700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f757806af60 0x7f757806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:48.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.875+0000 7f7599512700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f757806af60 0x7f757806d420 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f75840090b0 tx=0x7f758401e040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:48.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:48.879+0000 7f7589ffb700 1 -- 172.21.15.89:0/103678482 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f759008ae60 con 0x7f7594160ec0 2023-11-24T06:20:49.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.151+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_info"]} v 0) v1 -- 0x7f7594067f60 con 0x7f7594160ec0 2023-11-24T06:20:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.387+0000 7f7589ffb700 1 -- 172.21.15.89:0/103678482 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_info"]}]=0 v139) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7590019020 con 0x7f7594160ec0 2023-11-24T06:20:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f757806af60 msgr2=0x7f757806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f757806af60 0x7f757806d420 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f75840090b0 tx=0x7f758401e040 comp rx=0 tx=0).stop 2023-11-24T06:20:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7594160ec0 msgr2=0x7f7594161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7594160ec0 0x7f7594161340 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f759000e800 tx=0x7f759000eb10 comp rx=0 tx=0).stop 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 shutdown_connections 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f757806af60 0x7f757806d420 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75940cdaa0 0x7f75940cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7580003a00 0x7f75940cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 --2- 172.21.15.89:0/103678482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7594160ec0 0x7f7594161340 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 >> 172.21.15.89:0/103678482 conn(0x7f7594071ac0 msgr2=0x7f75940b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 shutdown_connections 2023-11-24T06:20:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.390+0000 7f759b776700 1 -- 172.21.15.89:0/103678482 wait complete. 2023-11-24T06:20:49.405 INFO:tasks.cephfs_test_runner:Tests the 'fs volume info' command ... ok 2023-11-24T06:20:49.405 INFO:tasks.cephfs_test_runner:Starting test: test_volume_info_pending_subvol_deletions (tasks.cephfs.test_volumes.TestVolumes) Tests the pending_subvolume_deletions in 'fs volume info' command 2023-11-24T06:20:49.405 INFO:tasks.cephfs_test_runner:test_volume_info_pending_subvol_deletions (tasks.cephfs.test_volumes.TestVolumes) 2023-11-24T06:20:49.406 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.529+0000 7f8837858700 1 Processor -- start 2023-11-24T06:20:49.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.530+0000 7f8837858700 1 -- start start 2023-11-24T06:20:49.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.530+0000 7f8837858700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f883015dfc0 0x7f88300d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.530+0000 7f8837858700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88300b2fc0 0x7f88300d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.530+0000 7f8837858700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88300b2a70 0x7f8830161b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.530+0000 7f8837858700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88300d2d10 con 0x7f88300b2fc0 2023-11-24T06:20:49.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.530+0000 7f8837858700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88300cefd0 con 0x7f883015dfc0 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.530+0000 7f8837858700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88300cf2b0 con 0x7f88300b2a70 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.530+0000 7f8834df3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88300b2fc0 0x7f88300d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8834df3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88300b2fc0 0x7f88300d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38680/0 (socket says 172.21.15.89:38680) 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8834df3700 1 -- 172.21.15.89:0/1769666424 learned_addr learned my addr 172.21.15.89:0/1769666424 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8834df3700 1 -- 172.21.15.89:0/1769666424 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88300b2a70 msgr2=0x7f8830161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8834df3700 1 --2- 172.21.15.89:0/1769666424 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88300b2a70 0x7f8830161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f88355f4700 1 --2- 172.21.15.89:0/1769666424 >> v2:172.21.15.123:3300/0 conn(0x7f883015dfc0 0x7f88300d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8834df3700 1 -- 172.21.15.89:0/1769666424 >> v2:172.21.15.123:3300/0 conn(0x7f883015dfc0 msgr2=0x7f88300d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8834df3700 1 --2- 172.21.15.89:0/1769666424 >> v2:172.21.15.123:3300/0 conn(0x7f883015dfc0 0x7f88300d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8834df3700 1 -- 172.21.15.89:0/1769666424 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88300d2fb0 con 0x7f88300b2fc0 2023-11-24T06:20:49.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f88355f4700 1 --2- 172.21.15.89:0/1769666424 >> v2:172.21.15.123:3300/0 conn(0x7f883015dfc0 0x7f88300d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8834df3700 1 --2- 172.21.15.89:0/1769666424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88300b2fc0 0x7f88300d13a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f88300d2230 tx=0x7f882c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e724cfb6a70acbc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8827fff700 1 -- 172.21.15.89:0/1769666424 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c00b870 con 0x7f88300b2fc0 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8827fff700 1 -- 172.21.15.89:0/1769666424 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c00b9d0 con 0x7f88300b2fc0 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.531+0000 7f8827fff700 1 -- 172.21.15.89:0/1769666424 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c00bd60 con 0x7f88300b2fc0 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 -- 172.21.15.89:0/1769666424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88300b2fc0 msgr2=0x7f88300d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 --2- 172.21.15.89:0/1769666424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88300b2fc0 0x7f88300d13a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f88300d2230 tx=0x7f882c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 -- 172.21.15.89:0/1769666424 shutdown_connections 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 --2- 172.21.15.89:0/1769666424 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88300b2a70 0x7f8830161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 --2- 172.21.15.89:0/1769666424 >> v2:172.21.15.123:3300/0 conn(0x7f883015dfc0 0x7f88300d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 --2- 172.21.15.89:0/1769666424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88300b2fc0 0x7f88300d13a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 -- 172.21.15.89:0/1769666424 >> 172.21.15.89:0/1769666424 conn(0x7f883006b410 msgr2=0x7f883006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:49.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 -- 172.21.15.89:0/1769666424 shutdown_connections 2023-11-24T06:20:49.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.532+0000 7f8837858700 1 -- 172.21.15.89:0/1769666424 wait complete. 2023-11-24T06:20:49.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.533+0000 7f8837858700 1 Processor -- start 2023-11-24T06:20:49.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.533+0000 7f8837858700 1 -- start start 2023-11-24T06:20:49.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8837858700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88300b2a70 0x7f88300ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8837858700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88300b2fc0 0x7f88300ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8837858700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883015dfc0 0x7f88300c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8837858700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88300cd5d0 con 0x7f883015dfc0 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8837858700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88300cd8b0 con 0x7f88300b2fc0 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8837858700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88300cdb90 con 0x7f88300b2a70 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883015dfc0 0x7f88300c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883015dfc0 0x7f88300c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38688/0 (socket says 172.21.15.89:38688) 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 -- 172.21.15.89:0/418791581 learned_addr learned my addr 172.21.15.89:0/418791581 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 -- 172.21.15.89:0/418791581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88300b2a70 msgr2=0x7f88300ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88300b2a70 0x7f88300ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 -- 172.21.15.89:0/418791581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88300b2fc0 msgr2=0x7f88300ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88300b2fc0 0x7f88300ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 -- 172.21.15.89:0/418791581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c009550 con 0x7f883015dfc0 2023-11-24T06:20:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8835df5700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883015dfc0 0x7f88300c96d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f882800b970 tx=0x7f882800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.534+0000 7f8825ffb700 1 -- 172.21.15.89:0/418791581 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882800c9b0 con 0x7f883015dfc0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.535+0000 7f8825ffb700 1 -- 172.21.15.89:0/418791581 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882800cb30 con 0x7f883015dfc0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.535+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88300cde70 con 0x7f883015dfc0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.535+0000 7f8825ffb700 1 -- 172.21.15.89:0/418791581 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8828014440 con 0x7f883015dfc0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.535+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88300ab790 con 0x7f883015dfc0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.536+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8830067f60 con 0x7f883015dfc0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.536+0000 7f8825ffb700 1 -- 172.21.15.89:0/418791581 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f882801d3b0 con 0x7f883015dfc0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.537+0000 7f8825ffb700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f881406af60 0x7f881406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.537+0000 7f88355f4700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f881406af60 0x7f881406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.537+0000 7f8825ffb700 1 -- 172.21.15.89:0/418791581 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8828089290 con 0x7f883015dfc0 2023-11-24T06:20:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.537+0000 7f88355f4700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f881406af60 0x7f881406d420 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8820007920 tx=0x7f8820008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.541+0000 7f8825ffb700 1 -- 172.21.15.89:0/418791581 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88280544c0 con 0x7f883015dfc0 2023-11-24T06:20:49.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.738+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88300ced10 con 0x7f883015dfc0 2023-11-24T06:20:49.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.739+0000 7f8825ffb700 1 -- 172.21.15.89:0/418791581 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+12950 (secure 0 0 0) 0x7f8828019090 con 0x7f883015dfc0 2023-11-24T06:20:49.742 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:49.742 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":56,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:48.381731+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/1341821445":"2023-11-25T06:20:45.367218+0000","172.21.15.89:6835/1341821445":"2023-11-25T06:20:45.367218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f881406af60 msgr2=0x7f881406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f881406af60 0x7f881406d420 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8820007920 tx=0x7f8820008040 comp rx=0 tx=0).stop 2023-11-24T06:20:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883015dfc0 msgr2=0x7f88300c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883015dfc0 0x7f88300c96d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f882800b970 tx=0x7f882800bd30 comp rx=0 tx=0).stop 2023-11-24T06:20:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 shutdown_connections 2023-11-24T06:20:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f883015dfc0 0x7f88300c96d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88300b2a70 0x7f88300ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f881406af60 0x7f881406d420 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 --2- 172.21.15.89:0/418791581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88300b2fc0 0x7f88300ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 >> 172.21.15.89:0/418791581 conn(0x7f883006b410 msgr2=0x7f88300b8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 shutdown_connections 2023-11-24T06:20:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.741+0000 7f8837858700 1 -- 172.21.15.89:0/418791581 wait complete. 2023-11-24T06:20:49.757 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:49.757 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:49.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.878+0000 7f1a6253e700 1 Processor -- start 2023-11-24T06:20:49.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.878+0000 7f1a6253e700 1 -- start start 2023-11-24T06:20:49.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a6253e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1a5c15fa30 0x7f1a5c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a6253e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1a5c0cfea0 0x7f1a5c0d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a6253e700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0b30f0 con 0x7f1a5c1455a0 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a6253e700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d0850 con 0x7f1a5c0cfea0 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a6253e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0ce670 con 0x7f1a5c15fa30 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a60adb700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1a5c0cfea0 0x7f1a5c0d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a5bfff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1a5c1455a0 0x7f1a5c15ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42198/0 (socket says 172.21.15.89:42198) 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a5bfff700 1 -- 172.21.15.89:0/1539926002 learned_addr learned my addr 172.21.15.89:0/1539926002 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a5b7fe700 1 --2- 172.21.15.89:0/1539926002 >> v2:172.21.15.123:3300/0 conn(0x7f1a5c15fa30 0x7f1a5c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a60adb700 1 -- 172.21.15.89:0/1539926002 >> v2:172.21.15.123:3300/0 conn(0x7f1a5c15fa30 msgr2=0x7f1a5c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a60adb700 1 --2- 172.21.15.89:0/1539926002 >> v2:172.21.15.123:3300/0 conn(0x7f1a5c15fa30 0x7f1a5c0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a60adb700 1 -- 172.21.15.89:0/1539926002 >> v1:172.21.15.89:6789/0 conn(0x7f1a5c1455a0 legacy=0x7f1a5c15ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a5affd700 1 -- 172.21.15.89:0/1539926002 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380107685 0 0) 0x7f1a5c0b30f0 con 0x7f1a5c1455a0 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a60adb700 1 -- 172.21.15.89:0/1539926002 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c160030 con 0x7f1a5c0cfea0 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.879+0000 7f1a5b7fe700 1 --2- 172.21.15.89:0/1539926002 >> v2:172.21.15.123:3300/0 conn(0x7f1a5c15fa30 0x7f1a5c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a60adb700 1 --2- 172.21.15.89:0/1539926002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c0cfea0 0x7f1a5c0d0310 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c072930 tx=0x7f1a5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77cfa67de471800 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a5affd700 1 -- 172.21.15.89:0/1539926002 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400e070 con 0x7f1a5c0cfea0 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a5affd700 1 -- 172.21.15.89:0/1539926002 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5400bbd0 con 0x7f1a5c0cfea0 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a5affd700 1 -- 172.21.15.89:0/1539926002 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54012430 con 0x7f1a5c0cfea0 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a6253e700 1 -- 172.21.15.89:0/1539926002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c0cfea0 msgr2=0x7f1a5c0d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1539926002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c0cfea0 0x7f1a5c0d0310 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c072930 tx=0x7f1a5400b330 comp rx=0 tx=0).stop 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a6253e700 1 -- 172.21.15.89:0/1539926002 shutdown_connections 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1539926002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c0cfea0 0x7f1a5c0d0310 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1539926002 >> v2:172.21.15.123:3300/0 conn(0x7f1a5c15fa30 0x7f1a5c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a6253e700 1 -- 172.21.15.89:0/1539926002 >> 172.21.15.89:0/1539926002 conn(0x7f1a5c06b410 msgr2=0x7f1a5c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:49.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a6253e700 1 -- 172.21.15.89:0/1539926002 shutdown_connections 2023-11-24T06:20:49.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.880+0000 7f1a6253e700 1 -- 172.21.15.89:0/1539926002 wait complete. 2023-11-24T06:20:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.881+0000 7f1a6253e700 1 Processor -- start 2023-11-24T06:20:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.882+0000 7f1a6253e700 1 -- start start 2023-11-24T06:20:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.882+0000 7f1a6253e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a5c0cfea0 0x7f1a5c158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.882+0000 7f1a6253e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c1455a0 0x7f1a5c15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.882+0000 7f1a6253e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a5c15fa30 0x7f1a5c13d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.882+0000 7f1a6253e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c15c7d0 con 0x7f1a5c1455a0 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.882+0000 7f1a6253e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c1598c0 con 0x7f1a5c15fa30 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.882+0000 7f1a6253e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c158d40 con 0x7f1a5c0cfea0 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c1455a0 0x7f1a5c15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c1455a0 0x7f1a5c15adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38702/0 (socket says 172.21.15.89:38702) 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 -- 172.21.15.89:0/1239171518 learned_addr learned my addr 172.21.15.89:0/1239171518 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a60adb700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a5c15fa30 0x7f1a5c13d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5bfff700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a5c0cfea0 0x7f1a5c158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 -- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a5c0cfea0 msgr2=0x7f1a5c158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a5c0cfea0 0x7f1a5c158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 -- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a5c15fa30 msgr2=0x7f1a5c13d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a5c15fa30 0x7f1a5c13d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 -- 172.21.15.89:0/1239171518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a54009a70 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5bfff700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a5c0cfea0 0x7f1a5c158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a5b7fe700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c1455a0 0x7f1a5c15adb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00c7f0 tx=0x7f1a4c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a58ff9700 1 -- 172.21.15.89:0/1239171518 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c011800 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a58ff9700 1 -- 172.21.15.89:0/1239171518 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c011960 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c159020 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a58ff9700 1 -- 172.21.15.89:0/1239171518 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c011d10 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.883+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c1595e0 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.884+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c068480 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.884+0000 7f1a58ff9700 1 -- 172.21.15.89:0/1239171518 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f1a4c018360 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.885+0000 7f1a58ff9700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1a4806af60 0x7f1a4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.885+0000 7f1a58ff9700 1 -- 172.21.15.89:0/1239171518 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1a4c0893c0 con 0x7f1a5c1455a0 2023-11-24T06:20:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.885+0000 7f1a5bfff700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1a4806af60 0x7f1a4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:49.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.885+0000 7f1a5bfff700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1a4806af60 0x7f1a4806d420 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1a50000f60 tx=0x7f1a50009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:49.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:49.889+0000 7f1a58ff9700 1 -- 172.21.15.89:0/1239171518 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a4c054670 con 0x7f1a5c1455a0 2023-11-24T06:20:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.080+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a5c068fb0 con 0x7f1a5c1455a0 2023-11-24T06:20:50.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.080+0000 7f1a58ff9700 1 -- 172.21.15.89:0/1239171518 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a4c057c90 con 0x7f1a5c1455a0 2023-11-24T06:20:50.083 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1a4806af60 msgr2=0x7f1a4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1a4806af60 0x7f1a4806d420 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1a50000f60 tx=0x7f1a50009040 comp rx=0 tx=0).stop 2023-11-24T06:20:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c1455a0 msgr2=0x7f1a5c15adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c1455a0 0x7f1a5c15adb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00c7f0 tx=0x7f1a4c00cbb0 comp rx=0 tx=0).stop 2023-11-24T06:20:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 shutdown_connections 2023-11-24T06:20:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1a4806af60 0x7f1a4806d420 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a5c15fa30 0x7f1a5c13d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a5c0cfea0 0x7f1a5c158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 --2- 172.21.15.89:0/1239171518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a5c1455a0 0x7f1a5c15adb0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 >> 172.21.15.89:0/1239171518 conn(0x7f1a5c06b410 msgr2=0x7f1a5c06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 shutdown_connections 2023-11-24T06:20:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.082+0000 7f1a6253e700 1 -- 172.21.15.89:0/1239171518 wait complete. 2023-11-24T06:20:50.097 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_info_pending_subvol_deletions' 2023-11-24T06:20:50.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.215+0000 7f1742927700 1 Processor -- start 2023-11-24T06:20:50.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.215+0000 7f1742927700 1 -- start start 2023-11-24T06:20:50.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f1742927700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f173c0adf00 0x7f173c06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:50.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f1742927700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173c0af950 con 0x7f173c1446a0 2023-11-24T06:20:50.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f1742927700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173c0afc30 con 0x7f173c06c540 2023-11-24T06:20:50.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f1742927700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173c06c9f0 con 0x7f173c0adf00 2023-11-24T06:20:50.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173bfff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f173c1446a0 0x7f173c0b5ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42212/0 (socket says 172.21.15.89:42212) 2023-11-24T06:20:50.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173bfff700 1 -- 172.21.15.89:0/3999108243 learned_addr learned my addr 172.21.15.89:0/3999108243 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:50.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173b7fe700 1 --2- 172.21.15.89:0/3999108243 >> v2:172.21.15.89:3300/0 conn(0x7f173c0adf00 0x7f173c06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173affd700 1 -- 172.21.15.89:0/3999108243 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005070743 0 0) 0x7f173c0af950 con 0x7f173c1446a0 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173affd700 1 -- 172.21.15.89:0/3999108243 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17240035b0 con 0x7f173c1446a0 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173b7fe700 1 -- 172.21.15.89:0/3999108243 >> v1:172.21.15.123:6789/0 conn(0x7f173c06c540 legacy=0x7f173c0b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173b7fe700 1 -- 172.21.15.89:0/3999108243 >> v1:172.21.15.89:6789/0 conn(0x7f173c1446a0 legacy=0x7f173c0b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173b7fe700 1 -- 172.21.15.89:0/3999108243 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c0aff10 con 0x7f173c0adf00 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173b7fe700 1 --2- 172.21.15.89:0/3999108243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c0adf00 0x7f173c06c000 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1730009840 tx=0x7f1730009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7764a491038a412 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.216+0000 7f173affd700 1 -- 172.21.15.89:0/3999108243 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730011830 con 0x7f173c0adf00 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f173affd700 1 -- 172.21.15.89:0/3999108243 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1730011990 con 0x7f173c0adf00 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f173affd700 1 -- 172.21.15.89:0/3999108243 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730011d20 con 0x7f173c0adf00 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f1742927700 1 -- 172.21.15.89:0/3999108243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c0adf00 msgr2=0x7f173c06c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f1742927700 1 --2- 172.21.15.89:0/3999108243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c0adf00 0x7f173c06c000 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1730009840 tx=0x7f1730009b50 comp rx=0 tx=0).stop 2023-11-24T06:20:50.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f1742927700 1 -- 172.21.15.89:0/3999108243 shutdown_connections 2023-11-24T06:20:50.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f1742927700 1 --2- 172.21.15.89:0/3999108243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c0adf00 0x7f173c06c000 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:50.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f1742927700 1 -- 172.21.15.89:0/3999108243 >> 172.21.15.89:0/3999108243 conn(0x7f173c071b90 msgr2=0x7f173c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:50.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f1742927700 1 -- 172.21.15.89:0/3999108243 shutdown_connections 2023-11-24T06:20:50.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.217+0000 7f1742927700 1 -- 172.21.15.89:0/3999108243 wait complete. 2023-11-24T06:20:50.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.219+0000 7f1742927700 1 Processor -- start 2023-11-24T06:20:50.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.219+0000 7f1742927700 1 -- start start 2023-11-24T06:20:50.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1742927700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c06c540 0x7f173c13b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:50.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1742927700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173c0adf00 0x7f173c158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:50.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1742927700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f173c1446a0 0x7f173c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:50.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1742927700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c159cd0 con 0x7f173c06c540 2023-11-24T06:20:50.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1742927700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c15b340 con 0x7f173c1446a0 2023-11-24T06:20:50.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1742927700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173c13b7e0 con 0x7f173c0adf00 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c06c540 0x7f173c13b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c06c540 0x7f173c13b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38714/0 (socket says 172.21.15.89:38714) 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 -- 172.21.15.89:0/4056113459 learned_addr learned my addr 172.21.15.89:0/4056113459 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 -- 172.21.15.89:0/4056113459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173c0adf00 msgr2=0x7f173c158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173c0adf00 0x7f173c158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 -- 172.21.15.89:0/4056113459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f173c1446a0 msgr2=0x7f173c0cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f173c1446a0 0x7f173c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 -- 172.21.15.89:0/4056113459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17300094a0 con 0x7f173c06c540 2023-11-24T06:20:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f173bfff700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c06c540 0x7f173c13b2a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c4d0 tx=0x7f172c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1738ff9700 1 -- 172.21.15.89:0/4056113459 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c005d70 con 0x7f173c06c540 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1738ff9700 1 -- 172.21.15.89:0/4056113459 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172c0070d0 con 0x7f173c06c540 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173c159190 con 0x7f173c06c540 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1738ff9700 1 -- 172.21.15.89:0/4056113459 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c007460 con 0x7f173c06c540 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.220+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173c15b830 con 0x7f173c06c540 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.221+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f173c0753f0 con 0x7f173c06c540 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.222+0000 7f1738ff9700 1 -- 172.21.15.89:0/4056113459 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f172c0075c0 con 0x7f173c06c540 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.222+0000 7f1738ff9700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f172806af60 0x7f172806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.222+0000 7f1738ff9700 1 -- 172.21.15.89:0/4056113459 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f172c088830 con 0x7f173c06c540 2023-11-24T06:20:50.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.222+0000 7f173b7fe700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f172806af60 0x7f172806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:50.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.222+0000 7f173b7fe700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f172806af60 0x7f172806d420 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1730009840 tx=0x7f1730010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:50.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.227+0000 7f1738ff9700 1 -- 172.21.15.89:0/4056113459 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f172c053bf0 con 0x7f173c06c540 2023-11-24T06:20:50.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:50.497+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_info_pending_subvol_deletions"]} v 0) v1 -- 0x7f173c067f60 con 0x7f173c06c540 2023-11-24T06:20:51.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.394+0000 7f1738ff9700 1 -- 172.21.15.89:0/4056113459 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_volume_info_pending_subvol_deletions"]}]=0 v141) v1 ==== 161+0+0 (secure 0 0 0) 0x7f172c0170e0 con 0x7f173c06c540 2023-11-24T06:20:51.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.396+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f172806af60 msgr2=0x7f172806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:51.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.396+0000 7f1742927700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f172806af60 0x7f172806d420 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1730009840 tx=0x7f1730010000 comp rx=0 tx=0).stop 2023-11-24T06:20:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.396+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c06c540 msgr2=0x7f173c13b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.396+0000 7f1742927700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c06c540 0x7f173c13b2a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c4d0 tx=0x7f172c00c890 comp rx=0 tx=0).stop 2023-11-24T06:20:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.397+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 shutdown_connections 2023-11-24T06:20:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.397+0000 7f1742927700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f172806af60 0x7f172806d420 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.397+0000 7f1742927700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f173c1446a0 0x7f173c0cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.397+0000 7f1742927700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f173c06c540 0x7f173c13b2a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.397+0000 7f1742927700 1 --2- 172.21.15.89:0/4056113459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173c0adf00 0x7f173c158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.397+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 >> 172.21.15.89:0/4056113459 conn(0x7f173c071b90 msgr2=0x7f173c0b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:51.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.397+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 shutdown_connections 2023-11-24T06:20:51.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.397+0000 7f1742927700 1 -- 172.21.15.89:0/4056113459 wait complete. 2023-11-24T06:20:51.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T06:20:51.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.540+0000 7f91ae7f5700 1 Processor -- start 2023-11-24T06:20:51.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.541+0000 7f91ae7f5700 1 -- start start 2023-11-24T06:20:51.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.541+0000 7f91ae7f5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 0x7f91a80cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.541+0000 7f91ae7f5700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f91a80c5100 0x7f91a80b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.541+0000 7f91ae7f5700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a815e750 con 0x7f91a80cf440 2023-11-24T06:20:51.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.541+0000 7f91ae7f5700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a8099c30 con 0x7f91a80c5100 2023-11-24T06:20:51.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.541+0000 7f91ae7f5700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80cea70 con 0x7f91a815e370 2023-11-24T06:20:51.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a77fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f91a80cf440 0x7f91a80cde10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42228/0 (socket says 172.21.15.89:42228) 2023-11-24T06:20:51.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a77fe700 1 -- 172.21.15.89:0/1516152544 learned_addr learned my addr 172.21.15.89:0/1516152544 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:51.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159316009 0 0) 0x7f91a815e750 con 0x7f91a80cf440 2023-11-24T06:20:51.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91940035b0 con 0x7f91a80cf440 2023-11-24T06:20:51.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91acd92700 1 --2- 172.21.15.89:0/1516152544 >> v2:172.21.15.123:3300/0 conn(0x7f91a80c5100 0x7f91a80b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a7fff700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 0x7f91a80cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439903629 0 0) 0x7f91940035b0 con 0x7f91a80cf440 2023-11-24T06:20:51.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 msgr2=0x7f91a80cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:51.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 0x7f91a80cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 >> v2:172.21.15.123:3300/0 conn(0x7f91a80c5100 msgr2=0x7f91a80b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:51.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 --2- 172.21.15.89:0/1516152544 >> v2:172.21.15.123:3300/0 conn(0x7f91a80c5100 0x7f91a80b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a8099e20 con 0x7f91a80cf440 2023-11-24T06:20:51.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f9198001670 con 0x7f91a80cf440 2023-11-24T06:20:51.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91acd92700 1 --2- 172.21.15.89:0/1516152544 >> v2:172.21.15.123:3300/0 conn(0x7f91a80c5100 0x7f91a80b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:51.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 >> v1:172.21.15.89:6789/0 conn(0x7f91a80cf440 legacy=0x7f91a80cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:51.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f9194003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9194004850 0x7f9194004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 0x7f9194005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80cea70 con 0x7f9194003a00 2023-11-24T06:20:51.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919400ba40 con 0x7f9194004850 2023-11-24T06:20:51.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919400bc00 con 0x7f91a815e370 2023-11-24T06:20:51.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a7fff700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f9194003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91acd92700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 0x7f9194005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a77fe700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9194004850 0x7f9194004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a7fff700 1 -- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 msgr2=0x7f9194005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:51.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a7fff700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 0x7f9194005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a7fff700 1 -- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9194004850 msgr2=0x7f9194004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:51.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a7fff700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9194004850 0x7f9194004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a7fff700 1 -- 172.21.15.89:0/1516152544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91940035b0 con 0x7f9194003a00 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91a77fe700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9194004850 0x7f9194004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.542+0000 7f91acd92700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a815e370 0x7f9194005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91a7fff700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f9194003e20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9190009860 tx=0x7f9194005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a20bf7daf9ce00 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919000c070 con 0x7f9194003a00 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190013610 con 0x7f9194003a00 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91a6ffd700 1 -- 172.21.15.89:0/1516152544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91900139a0 con 0x7f9194003a00 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91ae7f5700 1 -- 172.21.15.89:0/1516152544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 msgr2=0x7f9194003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91ae7f5700 1 --2- 172.21.15.89:0/1516152544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f9194003e20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9190009860 tx=0x7f9194005880 comp rx=0 tx=0).stop 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91acd92700 1 -- 172.21.15.89:0/1516152544 reap_dead start 2023-11-24T06:20:51.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91ae7f5700 1 -- 172.21.15.89:0/1516152544 shutdown_connections 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91ae7f5700 1 -- 172.21.15.89:0/1516152544 >> 172.21.15.89:0/1516152544 conn(0x7f91a80b1e90 msgr2=0x7f91a80b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91ae7f5700 1 -- 172.21.15.89:0/1516152544 shutdown_connections 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.543+0000 7f91ae7f5700 1 -- 172.21.15.89:0/1516152544 wait complete. 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.545+0000 7f91ae7f5700 1 Processor -- start 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.545+0000 7f91ae7f5700 1 -- start start 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91ae7f5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f91a8151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91ae7f5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91a80b09e0 0x7f91a80b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91ae7f5700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a80b0e60 0x7f91a80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91ae7f5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a81f80d0 con 0x7f9194003a00 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91ae7f5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80af980 con 0x7f91a80b09e0 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91ae7f5700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80afc20 con 0x7f91a80b0e60 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f91a8151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f91a8151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38734/0 (socket says 172.21.15.89:38734) 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 -- 172.21.15.89:0/465364085 learned_addr learned my addr 172.21.15.89:0/465364085 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a77fe700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91a80b09e0 0x7f91a80b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91acd92700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a80b0e60 0x7f91a80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 -- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a80b0e60 msgr2=0x7f91a80b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a80b0e60 0x7f91a80b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 -- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91a80b09e0 msgr2=0x7f91a80b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91a80b09e0 0x7f91a80b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 -- 172.21.15.89:0/465364085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a80aff00 con 0x7f9194003a00 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91acd92700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a80b0e60 0x7f91a80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a7fff700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f91a8151f60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9190016040 tx=0x7f91900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a4ff9700 1 -- 172.21.15.89:0/465364085 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190013a50 con 0x7f9194003a00 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a4ff9700 1 -- 172.21.15.89:0/465364085 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190013bb0 con 0x7f9194003a00 2023-11-24T06:20:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a80b01e0 con 0x7f9194003a00 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91a4ff9700 1 -- 172.21.15.89:0/465364085 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91900036a0 con 0x7f9194003a00 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.547+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a81524a0 con 0x7f9194003a00 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.548+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a8152630 con 0x7f9194003a00 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.548+0000 7f91a4ff9700 1 -- 172.21.15.89:0/465364085 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f9190003800 con 0x7f9194003a00 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.548+0000 7f91a4ff9700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f918806af60 0x7f918806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.549+0000 7f91a4ff9700 1 -- 172.21.15.89:0/465364085 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9190089d80 con 0x7f9194003a00 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.549+0000 7f91a77fe700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f918806af60 0x7f918806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.549+0000 7f91a77fe700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f918806af60 0x7f918806d420 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9198000fb0 tx=0x7f9198009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.554+0000 7f91a4ff9700 1 -- 172.21.15.89:0/465364085 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9190055170 con 0x7f9194003a00 2023-11-24T06:20:51.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.746+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f91a8068fb0 con 0x7f9194003a00 2023-11-24T06:20:51.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.750+0000 7f91a4ff9700 1 -- 172.21.15.89:0/465364085 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9190058790 con 0x7f9194003a00 2023-11-24T06:20:51.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f918806af60 msgr2=0x7f918806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:51.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f918806af60 0x7f918806d420 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9198000fb0 tx=0x7f9198009040 comp rx=0 tx=0).stop 2023-11-24T06:20:51.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 msgr2=0x7f91a8151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:51.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f91a8151f60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9190016040 tx=0x7f91900047c0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 shutdown_connections 2023-11-24T06:20:51.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91a80b0e60 0x7f91a80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9194003a00 0x7f91a8151f60 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f918806af60 0x7f918806d420 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 --2- 172.21.15.89:0/465364085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91a80b09e0 0x7f91a80b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:51.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 >> 172.21.15.89:0/465364085 conn(0x7f91a80b1e90 msgr2=0x7f91a80cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:51.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 shutdown_connections 2023-11-24T06:20:51.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:51.753+0000 7f91ae7f5700 1 -- 172.21.15.89:0/465364085 wait complete. 2023-11-24T06:20:51.768 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T06:20:51.768 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:20:51.812 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:20:51.862 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T06:20:51.862 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:51.863 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:51.880 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:20:51.880 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:51.880 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:51.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:20:51.901 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:20:51.901 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:51.937 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:20:51.937 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:20:51.937 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:20:51.963 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:20:51.963 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T06:20:51.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:20:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f39856fd700 1 Processor -- start 2023-11-24T06:20:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f39856fd700 1 -- start start 2023-11-24T06:20:52.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f39856fd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f398015e480 0x7f39800d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f39856fd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f39800d3610 0x7f39800b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f39856fd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 0x7f398006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f39856fd700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800b6300 con 0x7f398015e480 2023-11-24T06:20:52.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f39856fd700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800d2a70 con 0x7f39800d3610 2023-11-24T06:20:52.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f39856fd700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800d2d50 con 0x7f39800b5e90 2023-11-24T06:20:52.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f398015e480 0x7f39800d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f398015e480 0x7f39800d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38742/0 (socket says 172.21.15.89:38742) 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 -- 172.21.15.89:0/1358776431 learned_addr learned my addr 172.21.15.89:0/1358776431 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 -- 172.21.15.89:0/1358776431 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 msgr2=0x7f398006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 --2- 172.21.15.89:0/1358776431 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 0x7f398006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 -- 172.21.15.89:0/1358776431 >> v2:172.21.15.123:3300/0 conn(0x7f39800d3610 msgr2=0x7f39800b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 --2- 172.21.15.89:0/1358776431 >> v2:172.21.15.123:3300/0 conn(0x7f39800d3610 0x7f39800b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 -- 172.21.15.89:0/1358776431 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800d3030 con 0x7f398015e480 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397effd700 1 --2- 172.21.15.89:0/1358776431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f39800d2100 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f398008c8e0 tx=0x7f3970009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4384d0f47b7a8f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397dffb700 1 -- 172.21.15.89:0/1358776431 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397000b870 con 0x7f398015e480 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397dffb700 1 -- 172.21.15.89:0/1358776431 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397000b9d0 con 0x7f398015e480 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.095+0000 7f397dffb700 1 -- 172.21.15.89:0/1358776431 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397000bd60 con 0x7f398015e480 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 -- 172.21.15.89:0/1358776431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 msgr2=0x7f39800d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 --2- 172.21.15.89:0/1358776431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f39800d2100 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f398008c8e0 tx=0x7f3970009bd0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 -- 172.21.15.89:0/1358776431 shutdown_connections 2023-11-24T06:20:52.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 --2- 172.21.15.89:0/1358776431 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 0x7f398006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 --2- 172.21.15.89:0/1358776431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f39800d2100 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 --2- 172.21.15.89:0/1358776431 >> v2:172.21.15.123:3300/0 conn(0x7f39800d3610 0x7f39800b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 -- 172.21.15.89:0/1358776431 >> 172.21.15.89:0/1358776431 conn(0x7f39800d0950 msgr2=0x7f39800d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:52.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 -- 172.21.15.89:0/1358776431 shutdown_connections 2023-11-24T06:20:52.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.096+0000 7f39856fd700 1 -- 172.21.15.89:0/1358776431 wait complete. 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.097+0000 7f39856fd700 1 Processor -- start 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.097+0000 7f39856fd700 1 -- start start 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f39856fd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 0x7f39801f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f39856fd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39800d3610 0x7f398015e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f39856fd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f3980160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f39856fd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398015fde0 con 0x7f398015e480 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f39856fd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398015f1d0 con 0x7f39800d3610 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f39856fd700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398015f4b0 con 0x7f39800b5e90 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f3980160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f3980160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38752/0 (socket says 172.21.15.89:38752) 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 -- 172.21.15.89:0/2916922367 learned_addr learned my addr 172.21.15.89:0/2916922367 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:52.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397e7fc700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39800d3610 0x7f398015e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397effd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 0x7f39801f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 -- 172.21.15.89:0/2916922367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 msgr2=0x7f39801f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 0x7f39801f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 -- 172.21.15.89:0/2916922367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39800d3610 msgr2=0x7f398015e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39800d3610 0x7f398015e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 -- 172.21.15.89:0/2916922367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970009550 con 0x7f398015e480 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.098+0000 7f397f7fe700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f3980160270 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f397800b9e0 tx=0x7f397800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.099+0000 7f39677fe700 1 -- 172.21.15.89:0/2916922367 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397800d820 con 0x7f398015e480 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.099+0000 7f39677fe700 1 -- 172.21.15.89:0/2916922367 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397800d9a0 con 0x7f398015e480 2023-11-24T06:20:52.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.099+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398015f790 con 0x7f398015e480 2023-11-24T06:20:52.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.099+0000 7f39677fe700 1 -- 172.21.15.89:0/2916922367 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397800dd50 con 0x7f398015e480 2023-11-24T06:20:52.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.099+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39801f7640 con 0x7f398015e480 2023-11-24T06:20:52.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.100+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398015ff50 con 0x7f398015e480 2023-11-24T06:20:52.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.100+0000 7f39677fe700 1 -- 172.21.15.89:0/2916922367 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f3978015420 con 0x7f398015e480 2023-11-24T06:20:52.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.100+0000 7f39677fe700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f396c06af60 0x7f396c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.100+0000 7f39677fe700 1 -- 172.21.15.89:0/2916922367 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3978011070 con 0x7f398015e480 2023-11-24T06:20:52.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.100+0000 7f397effd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f396c06af60 0x7f396c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.101+0000 7f397effd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f396c06af60 0x7f396c06d420 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3970014040 tx=0x7f3970008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:52.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.106+0000 7f39677fe700 1 -- 172.21.15.89:0/2916922367 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f397804ee90 con 0x7f398015e480 2023-11-24T06:20:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.340+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39800751e0 con 0x7f398015e480 2023-11-24T06:20:52.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.340+0000 7f39677fe700 1 -- 172.21.15.89:0/2916922367 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f3978054450 con 0x7f398015e480 2023-11-24T06:20:52.342 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:52.343 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":34,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2011089843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2011089843},{"type":"v1","addr":"172.21.15.89:6836","nonce":2011089843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4955,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/2182962276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2182962276},{"type":"v1","addr":"172.21.15.123:6834","nonce":2182962276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":4958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2982219407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2982219407},{"type":"v1","addr":"172.21.15.123:6835","nonce":2982219407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3963190568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3963190568},{"type":"v1","addr":"172.21.15.89:6837","nonce":3963190568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[]} 2023-11-24T06:20:52.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f396c06af60 msgr2=0x7f396c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f396c06af60 0x7f396c06d420 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3970014040 tx=0x7f3970008030 comp rx=0 tx=0).stop 2023-11-24T06:20:52.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 msgr2=0x7f3980160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f3980160270 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f397800b9e0 tx=0x7f397800bcf0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 shutdown_connections 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398015e480 0x7f3980160270 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800b5e90 0x7f39801f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f396c06af60 0x7f396c06d420 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 --2- 172.21.15.89:0/2916922367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39800d3610 0x7f398015e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 >> 172.21.15.89:0/2916922367 conn(0x7f39800d0950 msgr2=0x7f39800b8120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 shutdown_connections 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.342+0000 7f39856fd700 1 -- 172.21.15.89:0/2916922367 wait complete. 2023-11-24T06:20:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 34 2023-11-24T06:20:52.357 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T06:20:52.357 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T06:20:52.357 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T06:20:52.357 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:20:52.357 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T06:20:52.357 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T06:20:52.358 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T06:20:52.358 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:20:52.358 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T06:20:52.358 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T06:20:52.358 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T06:20:52.358 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:20:52.358 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T06:20:52.358 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T06:20:52.359 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T06:20:52.359 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:20:52.366 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T06:20:52.366 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T06:20:52.369 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T06:20:52.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T06:20:52.370 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T06:20:52.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T06:20:52.371 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T06:20:52.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T06:20:52.409 INFO:tasks.ceph.mds.a:Started 2023-11-24T06:20:52.409 DEBUG:teuthology.parallel:result is None 2023-11-24T06:20:52.410 INFO:tasks.ceph.mds.c:Started 2023-11-24T06:20:52.410 DEBUG:teuthology.parallel:result is None 2023-11-24T06:20:52.413 INFO:tasks.ceph.mds.b:Started 2023-11-24T06:20:52.413 DEBUG:teuthology.parallel:result is None 2023-11-24T06:20:52.414 INFO:tasks.ceph.mds.d:Started 2023-11-24T06:20:52.414 DEBUG:teuthology.parallel:result is None 2023-11-24T06:20:52.414 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T06:20:52.488 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-11-24T06:20:52.488 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2023-11-24T06:20:52.498 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2023-11-24T06:20:52.498 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2023-11-24T06:20:52.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.539+0000 7fe88333e700 1 Processor -- start 2023-11-24T06:20:52.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.540+0000 7fe88333e700 1 -- start start 2023-11-24T06:20:52.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.540+0000 7fe88333e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c15b0b0 0x7fe87c0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.540+0000 7fe88333e700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c0b30b0 con 0x7fe87c1587e0 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.540+0000 7fe88333e700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c0b3390 con 0x7fe87c06eb70 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.540+0000 7fe88333e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c0b3d30 con 0x7fe87c15b0b0 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe8810da700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe87c1587e0 0x7fe87c06e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42282/0 (socket says 172.21.15.89:42282) 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe8810da700 1 -- 172.21.15.89:0/2963928351 learned_addr learned my addr 172.21.15.89:0/2963928351 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe8818db700 1 --2- 172.21.15.89:0/2963928351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c15b0b0 0x7fe87c0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe87bfff700 1 -- 172.21.15.89:0/2963928351 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761071805 0 0) 0x7fe87c0b30b0 con 0x7fe87c1587e0 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe87bfff700 1 -- 172.21.15.89:0/2963928351 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8640035b0 con 0x7fe87c1587e0 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe87bfff700 1 -- 172.21.15.89:0/2963928351 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187145662 0 0) 0x7fe87c0b3390 con 0x7fe87c06eb70 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe87bfff700 1 -- 172.21.15.89:0/2963928351 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c0b30b0 con 0x7fe87c06eb70 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe8818db700 1 -- 172.21.15.89:0/2963928351 >> v1:172.21.15.123:6789/0 conn(0x7fe87c06eb70 legacy=0x7fe87c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe8818db700 1 -- 172.21.15.89:0/2963928351 >> v1:172.21.15.89:6789/0 conn(0x7fe87c1587e0 legacy=0x7fe87c06e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe87bfff700 1 -- 172.21.15.89:0/2963928351 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1808144138 0 0) 0x7fe8640035b0 con 0x7fe87c1587e0 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe8818db700 1 -- 172.21.15.89:0/2963928351 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c0b3670 con 0x7fe87c15b0b0 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe8818db700 1 --2- 172.21.15.89:0/2963928351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c15b0b0 0x7fe87c0b2a40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe874009870 tx=0x7fe874009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfcf86ab43ea15ba server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe87bfff700 1 -- 172.21.15.89:0/2963928351 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0019f0 con 0x7fe87c15b0b0 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe87bfff700 1 -- 172.21.15.89:0/2963928351 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe874004030 con 0x7fe87c15b0b0 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.541+0000 7fe87bfff700 1 -- 172.21.15.89:0/2963928351 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8740043c0 con 0x7fe87c15b0b0 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.542+0000 7fe88333e700 1 -- 172.21.15.89:0/2963928351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c15b0b0 msgr2=0x7fe87c0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.542+0000 7fe88333e700 1 --2- 172.21.15.89:0/2963928351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c15b0b0 0x7fe87c0b2a40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe874009870 tx=0x7fe874009b80 comp rx=0 tx=0).stop 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.542+0000 7fe88333e700 1 -- 172.21.15.89:0/2963928351 shutdown_connections 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.542+0000 7fe88333e700 1 --2- 172.21.15.89:0/2963928351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c15b0b0 0x7fe87c0b2a40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.542+0000 7fe88333e700 1 -- 172.21.15.89:0/2963928351 >> 172.21.15.89:0/2963928351 conn(0x7fe87c0b1740 msgr2=0x7fe87c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.542+0000 7fe88333e700 1 -- 172.21.15.89:0/2963928351 shutdown_connections 2023-11-24T06:20:52.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.542+0000 7fe88333e700 1 -- 172.21.15.89:0/2963928351 wait complete. 2023-11-24T06:20:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.543+0000 7fe88333e700 1 Processor -- start 2023-11-24T06:20:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.543+0000 7fe88333e700 1 -- start start 2023-11-24T06:20:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe88333e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe87c06eb70 0x7fe87c1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe88333e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c1587e0 0x7fe87c0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe88333e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15b0b0 0x7fe87c147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe88333e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c1523b0 con 0x7fe87c15b0b0 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe88333e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c152690 con 0x7fe87c06eb70 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe88333e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c152970 con 0x7fe87c1587e0 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15b0b0 0x7fe87c147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15b0b0 0x7fe87c147280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38788/0 (socket says 172.21.15.89:38788) 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 -- 172.21.15.89:0/3287074585 learned_addr learned my addr 172.21.15.89:0/3287074585 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8810da700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe87c06eb70 0x7fe87c1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8808d9700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c1587e0 0x7fe87c0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 -- 172.21.15.89:0/3287074585 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c1587e0 msgr2=0x7fe87c0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c1587e0 0x7fe87c0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 -- 172.21.15.89:0/3287074585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe87c06eb70 msgr2=0x7fe87c1f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe87c06eb70 0x7fe87c1f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 -- 172.21.15.89:0/3287074585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8740094a0 con 0x7fe87c15b0b0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.544+0000 7fe8818db700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15b0b0 0x7fe87c147280 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe8740097c0 tx=0x7fe874004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.545+0000 7fe879ffb700 1 -- 172.21.15.89:0/3287074585 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874003c10 con 0x7fe87c15b0b0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.545+0000 7fe879ffb700 1 -- 172.21.15.89:0/3287074585 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87401f070 con 0x7fe87c15b0b0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.545+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c152c50 con 0x7fe87c15b0b0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.545+0000 7fe879ffb700 1 -- 172.21.15.89:0/3287074585 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874004570 con 0x7fe87c15b0b0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.545+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c0af2e0 con 0x7fe87c15b0b0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.546+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c0751e0 con 0x7fe87c15b0b0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.546+0000 7fe879ffb700 1 -- 172.21.15.89:0/3287074585 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fe874021030 con 0x7fe87c15b0b0 2023-11-24T06:20:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.547+0000 7fe879ffb700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe86806af60 0x7fe86806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.547+0000 7fe879ffb700 1 -- 172.21.15.89:0/3287074585 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe87408a4a0 con 0x7fe87c15b0b0 2023-11-24T06:20:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.547+0000 7fe8810da700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe86806af60 0x7fe86806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.547+0000 7fe8810da700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe86806af60 0x7fe86806d420 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0012e0 tx=0x7fe86c000f40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.551+0000 7fe879ffb700 1 -- 172.21.15.89:0/3287074585 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8740558f0 con 0x7fe87c15b0b0 2023-11-24T06:20:52.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:52.756+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe87c06ceb0 con 0x7fe87c15b0b0 2023-11-24T06:20:53.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.413+0000 7fe879ffb700 1 -- 172.21.15.89:0/3287074585 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v58) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe874058f10 con 0x7fe87c15b0b0 2023-11-24T06:20:53.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe86806af60 msgr2=0x7fe86806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe86806af60 0x7fe86806d420 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0012e0 tx=0x7fe86c000f40 comp rx=0 tx=0).stop 2023-11-24T06:20:53.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15b0b0 msgr2=0x7fe87c147280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15b0b0 0x7fe87c147280 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe8740097c0 tx=0x7fe874004750 comp rx=0 tx=0).stop 2023-11-24T06:20:53.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 shutdown_connections 2023-11-24T06:20:53.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe87c15b0b0 0x7fe87c147280 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe86806af60 0x7fe86806d420 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe87c06eb70 0x7fe87c1f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 --2- 172.21.15.89:0/3287074585 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe87c1587e0 0x7fe87c0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.417+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 >> 172.21.15.89:0/3287074585 conn(0x7fe87c0b1740 msgr2=0x7fe87c0af600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:53.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.418+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 shutdown_connections 2023-11-24T06:20:53.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.418+0000 7fe88333e700 1 -- 172.21.15.89:0/3287074585 wait complete. 2023-11-24T06:20:53.422 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2023-11-24T06:20:53.436 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T06:20:53.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.560+0000 7ffa51436700 1 Processor -- start 2023-11-24T06:20:53.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.560+0000 7ffa51436700 1 -- start start 2023-11-24T06:20:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa51436700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffa4c15e1e0 0x7ffa4c0d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa51436700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffa4c0d3a00 0x7ffa4c0d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa51436700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0d1200 con 0x7ffa4c0d0db0 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa51436700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c0d3150 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa51436700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c15e5c0 con 0x7ffa4c0d3a00 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4affd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffa4c15e1e0 0x7ffa4c0d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4affd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffa4c15e1e0 0x7ffa4c0d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38794/0 (socket says 172.21.15.89:38794) 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4affd700 1 -- 172.21.15.89:0/2655434351 learned_addr learned my addr 172.21.15.89:0/2655434351 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4a7fc700 1 --2- 172.21.15.89:0/2655434351 >> v2:172.21.15.123:3300/0 conn(0x7ffa4c0d3a00 0x7ffa4c0d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4affd700 1 -- 172.21.15.89:0/2655434351 >> v2:172.21.15.123:3300/0 conn(0x7ffa4c0d3a00 msgr2=0x7ffa4c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4affd700 1 --2- 172.21.15.89:0/2655434351 >> v2:172.21.15.123:3300/0 conn(0x7ffa4c0d3a00 0x7ffa4c0d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4affd700 1 -- 172.21.15.89:0/2655434351 >> v1:172.21.15.123:6789/0 conn(0x7ffa4c0d0db0 legacy=0x7ffa4c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4affd700 1 -- 172.21.15.89:0/2655434351 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0d20b0 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa4affd700 1 --2- 172.21.15.89:0/2655434351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c0d34c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c161ba0 tx=0x7ffa3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b84b924bf1c861c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.561+0000 7ffa49ffb700 1 -- 172.21.15.89:0/2655434351 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c004230 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa49ffb700 1 -- 172.21.15.89:0/2655434351 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3c004390 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa49ffb700 1 -- 172.21.15.89:0/2655434351 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0036a0 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa51436700 1 -- 172.21.15.89:0/2655434351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 msgr2=0x7ffa4c0d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa51436700 1 --2- 172.21.15.89:0/2655434351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c0d34c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c161ba0 tx=0x7ffa3c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa51436700 1 -- 172.21.15.89:0/2655434351 shutdown_connections 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa51436700 1 --2- 172.21.15.89:0/2655434351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c0d34c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa51436700 1 --2- 172.21.15.89:0/2655434351 >> v2:172.21.15.123:3300/0 conn(0x7ffa4c0d3a00 0x7ffa4c0d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa51436700 1 -- 172.21.15.89:0/2655434351 >> 172.21.15.89:0/2655434351 conn(0x7ffa4c0b5b50 msgr2=0x7ffa4c0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa51436700 1 -- 172.21.15.89:0/2655434351 shutdown_connections 2023-11-24T06:20:53.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.562+0000 7ffa51436700 1 -- 172.21.15.89:0/2655434351 wait complete. 2023-11-24T06:20:53.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.563+0000 7ffa51436700 1 Processor -- start 2023-11-24T06:20:53.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.563+0000 7ffa51436700 1 -- start start 2023-11-24T06:20:53.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa51436700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c0d0db0 0x7ffa4c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa51436700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa4c0d3a00 0x7ffa4c15fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa51436700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa51436700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c157670 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa51436700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c157930 con 0x7ffa4c0d0db0 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa51436700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c157c10 con 0x7ffa4c0d3a00 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa4b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa4b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38804/0 (socket says 172.21.15.89:38804) 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa4b7fe700 1 -- 172.21.15.89:0/2192654090 learned_addr learned my addr 172.21.15.89:0/2192654090 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa4a7fc700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa4c0d3a00 0x7ffa4c15fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.564+0000 7ffa4affd700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c0d0db0 0x7ffa4c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa4b7fe700 1 -- 172.21.15.89:0/2192654090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa4c0d3a00 msgr2=0x7ffa4c15fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa4b7fe700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa4c0d3a00 0x7ffa4c15fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa4b7fe700 1 -- 172.21.15.89:0/2192654090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c0d0db0 msgr2=0x7ffa4c156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa4b7fe700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c0d0db0 0x7ffa4c156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa4b7fe700 1 -- 172.21.15.89:0/2192654090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3c009550 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa4b7fe700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c152c80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400cc10 tx=0x7ffa4400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa337fe700 1 -- 172.21.15.89:0/2192654090 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4400aa60 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa337fe700 1 -- 172.21.15.89:0/2192654090 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4400abc0 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa337fe700 1 -- 172.21.15.89:0/2192654090 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa44012420 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c157ef0 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.565+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c157300 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.566+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c0751e0 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.567+0000 7ffa337fe700 1 -- 172.21.15.89:0/2192654090 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ffa44012580 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.567+0000 7ffa337fe700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffa3806af60 0x7ffa3806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.567+0000 7ffa337fe700 1 -- 172.21.15.89:0/2192654090 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffa44089650 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.567+0000 7ffa4affd700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffa3806af60 0x7ffa3806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.567+0000 7ffa4affd700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffa3806af60 0x7ffa3806d420 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009130 tx=0x7ffa3c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:53.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.571+0000 7ffa337fe700 1 -- 172.21.15.89:0/2192654090 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa440549d0 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.841+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ffa4c0d13a0 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.841+0000 7ffa337fe700 1 -- 172.21.15.89:0/2192654090 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ffa44057ff0 con 0x7ffa4c15e1e0 2023-11-24T06:20:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:53.843 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:20:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2023-11-24T06:20:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlZPhnExAAdSzrCYJj/BqJINrd+mxnog==", 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2023-11-24T06:20:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlAvtyHxAAiLxnKUSDAvkxFP6LHLBkSQ==", 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlrHqfGhAA7hKW48g57uUyHOiS4gQkOQ==", 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBl4PG2JhAAg8KlX9jNiQ/Ajvrm71BM8g==", 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQASQGBlLUQyBxAApB8r9XPrC9s6zsd2nPQFoQ==", 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVQGBlFgtPBhAAvSzzgIHzad8zv8vS3fcFQg==", 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAYQGBlNW1sBRAAtOtIYUOAa63AiLIS9EYCFA==", 2023-11-24T06:20:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAbQGBlZOIsBRAATWs4ayExofw/56n0ixjgDA==", 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAhQGBlGkIpCRAAWpMWRhHqDSEuyM+DAQLP8Q==", 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAkQGBli6LfCBAAH5LBKEpbvU48rAyWnyH0Zg==", 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAnQGBl6qpjCBAASFc7C3OIr9G6lSXCNyLk8A==", 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAqQGBlxmJdCBAANGRCESck0mrD1gVuToRRaw==", 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlwIfyKhAAMZcwOQGAm+XcH+mUKcTiUg==", 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2023-11-24T06:20:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlxNQDNhAApU+JeKCeMTDo61+wi3ynbw==", 2023-11-24T06:20:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:20:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:20:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:20:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2023-11-24T06:20:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBluCSzMhAA1hxy9u3cvwTZLG/Xg3kQWQ==", 2023-11-24T06:20:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:20:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:20:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:20:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlin9NORAAkdx6VeU4rwbgNfVPwwUXFg==", 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQANQGBl+ReMMhAARm9F19thbXVu988u1kapsw==", 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlj+EGEBAAFs6HBnMAVB9ZLdn7WrWpzQ==", 2023-11-24T06:20:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAOQGBlsrk+DBAAdgV15cObrTUfehK6bUY8DA==", 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2023-11-24T06:20:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2023-11-24T06:20:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-11-24T06:20:53.864 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:20:53.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffa3806af60 msgr2=0x7ffa3806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffa3806af60 0x7ffa3806d420 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009130 tx=0x7ffa3c008030 comp rx=0 tx=0).stop 2023-11-24T06:20:53.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 msgr2=0x7ffa4c152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c152c80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400cc10 tx=0x7ffa4400cf20 comp rx=0 tx=0).stop 2023-11-24T06:20:53.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 shutdown_connections 2023-11-24T06:20:53.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa4c15e1e0 0x7ffa4c152c80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ffa3806af60 0x7ffa3806d420 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa4c0d0db0 0x7ffa4c156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 --2- 172.21.15.89:0/2192654090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa4c0d3a00 0x7ffa4c15fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 >> 172.21.15.89:0/2192654090 conn(0x7ffa4c0b5b50 msgr2=0x7ffa4c0ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 shutdown_connections 2023-11-24T06:20:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.843+0000 7ffa51436700 1 -- 172.21.15.89:0/2192654090 wait complete. 2023-11-24T06:20:53.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.985+0000 7fefd2633700 1 Processor -- start 2023-11-24T06:20:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.985+0000 7fefd2633700 1 -- start start 2023-11-24T06:20:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.986+0000 7fefd2633700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc15e080 0x7fefcc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.986+0000 7fefd2633700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fefcc06c710 0x7fefcc0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.986+0000 7fefd2633700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefcc0ae430 con 0x7fefcc0af4d0 2023-11-24T06:20:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.986+0000 7fefd2633700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc0af0b0 con 0x7fefcc06c710 2023-11-24T06:20:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.986+0000 7fefd2633700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0b05a0 con 0x7fefcc15e080 2023-11-24T06:20:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.986+0000 7fefd0bd0700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fefcc06c710 0x7fefcc0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.986+0000 7fefcb7fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fefcc0af4d0 0x7fefcc06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42312/0 (socket says 172.21.15.89:42312) 2023-11-24T06:20:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.986+0000 7fefcb7fe700 1 -- 172.21.15.89:0/924097409 learned_addr learned my addr 172.21.15.89:0/924097409 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefcbfff700 1 --2- 172.21.15.89:0/924097409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc15e080 0x7fefcc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd0bd0700 1 -- 172.21.15.89:0/924097409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc15e080 msgr2=0x7fefcc0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd0bd0700 1 --2- 172.21.15.89:0/924097409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc15e080 0x7fefcc0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd0bd0700 1 -- 172.21.15.89:0/924097409 >> v1:172.21.15.89:6789/0 conn(0x7fefcc0af4d0 legacy=0x7fefcc06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd0bd0700 1 -- 172.21.15.89:0/924097409 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc0ae710 con 0x7fefcc06c710 2023-11-24T06:20:53.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefcaffd700 1 -- 172.21.15.89:0/924097409 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262855590 0 0) 0x7fefcc0ae430 con 0x7fefcc0af4d0 2023-11-24T06:20:53.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd0bd0700 1 --2- 172.21.15.89:0/924097409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc06c710 0x7fefcc0afb90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009dc0 tx=0x7fefc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c20d19b8cbfda03 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:53.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefcaffd700 1 -- 172.21.15.89:0/924097409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc400e070 con 0x7fefcc06c710 2023-11-24T06:20:53.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefcaffd700 1 -- 172.21.15.89:0/924097409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc400bb90 con 0x7fefcc06c710 2023-11-24T06:20:53.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefcaffd700 1 -- 172.21.15.89:0/924097409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4012430 con 0x7fefcc06c710 2023-11-24T06:20:53.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd2633700 1 -- 172.21.15.89:0/924097409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc06c710 msgr2=0x7fefcc0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd2633700 1 --2- 172.21.15.89:0/924097409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc06c710 0x7fefcc0afb90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009dc0 tx=0x7fefc400b330 comp rx=0 tx=0).stop 2023-11-24T06:20:53.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd2633700 1 -- 172.21.15.89:0/924097409 shutdown_connections 2023-11-24T06:20:53.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd2633700 1 --2- 172.21.15.89:0/924097409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc06c710 0x7fefcc0afb90 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd2633700 1 --2- 172.21.15.89:0/924097409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc15e080 0x7fefcc0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.987+0000 7fefd2633700 1 -- 172.21.15.89:0/924097409 >> 172.21.15.89:0/924097409 conn(0x7fefcc071b90 msgr2=0x7fefcc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:53.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.988+0000 7fefd2633700 1 -- 172.21.15.89:0/924097409 shutdown_connections 2023-11-24T06:20:53.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.988+0000 7fefd2633700 1 -- 172.21.15.89:0/924097409 wait complete. 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.989+0000 7fefd2633700 1 Processor -- start 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.989+0000 7fefd2633700 1 -- start start 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd2633700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefcc06c710 0x7fefcc150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd2633700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc0af4d0 0x7fefcc1545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd2633700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc15e080 0x7fefcc0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd2633700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc1f7a90 con 0x7fefcc15e080 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd2633700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc154b10 con 0x7fefcc06c710 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd2633700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc1f6f10 con 0x7fefcc0af4d0 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc15e080 0x7fefcc0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc15e080 0x7fefcc0c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38816/0 (socket says 172.21.15.89:38816) 2023-11-24T06:20:53.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 -- 172.21.15.89:0/3627129494 learned_addr learned my addr 172.21.15.89:0/3627129494 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefcbfff700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefcc06c710 0x7fefcc150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefcb7fe700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc0af4d0 0x7fefcc1545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 -- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc0af4d0 msgr2=0x7fefcc1545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc0af4d0 0x7fefcc1545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 -- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefcc06c710 msgr2=0x7fefcc150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefcc06c710 0x7fefcc150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 -- 172.21.15.89:0/3627129494 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4009a70 con 0x7fefcc15e080 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefcb7fe700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc0af4d0 0x7fefcc1545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd0bd0700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc15e080 0x7fefcc0c6f40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fefc4014040 tx=0x7fefc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefc8ff9700 1 -- 172.21.15.89:0/3627129494 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4012430 con 0x7fefcc15e080 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefc8ff9700 1 -- 172.21.15.89:0/3627129494 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc4020070 con 0x7fefcc15e080 2023-11-24T06:20:53.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc1f71f0 con 0x7fefcc15e080 2023-11-24T06:20:53.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefc8ff9700 1 -- 172.21.15.89:0/3627129494 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4004280 con 0x7fefcc15e080 2023-11-24T06:20:53.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.990+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc1f77b0 con 0x7fefcc15e080 2023-11-24T06:20:53.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.991+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc0753f0 con 0x7fefcc15e080 2023-11-24T06:20:53.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.992+0000 7fefc8ff9700 1 -- 172.21.15.89:0/3627129494 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fefc40043e0 con 0x7fefcc15e080 2023-11-24T06:20:53.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.992+0000 7fefc8ff9700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fefb806af60 0x7fefb806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:53.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.993+0000 7fefcbfff700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fefb806af60 0x7fefb806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:53.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.993+0000 7fefc8ff9700 1 -- 172.21.15.89:0/3627129494 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fefc4022070 con 0x7fefcc15e080 2023-11-24T06:20:53.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.993+0000 7fefcbfff700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fefb806af60 0x7fefb806d420 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fefc000b9b0 tx=0x7fefc0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:54.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:53.999+0000 7fefc8ff9700 1 -- 172.21.15.89:0/3627129494 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fefc4056d00 con 0x7fefcc15e080 2023-11-24T06:20:54.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.189+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefcc067f60 con 0x7fefcc15e080 2023-11-24T06:20:54.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.190+0000 7fefc8ff9700 1 -- 172.21.15.89:0/3627129494 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+12821 (secure 0 0 0) 0x7fefc401ae60 con 0x7fefcc15e080 2023-11-24T06:20:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:54.192 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":60,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:53.421507+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:54.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fefb806af60 msgr2=0x7fefb806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fefb806af60 0x7fefb806d420 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fefc000b9b0 tx=0x7fefc0008040 comp rx=0 tx=0).stop 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc15e080 msgr2=0x7fefcc0c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc15e080 0x7fefcc0c6f40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fefc4014040 tx=0x7fefc4004ad0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 shutdown_connections 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefcc15e080 0x7fefcc0c6f40 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fefb806af60 0x7fefb806d420 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefcc06c710 0x7fefcc150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 --2- 172.21.15.89:0/3627129494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fefcc0af4d0 0x7fefcc1545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 >> 172.21.15.89:0/3627129494 conn(0x7fefcc071b90 msgr2=0x7fefcc06e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 shutdown_connections 2023-11-24T06:20:54.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.192+0000 7fefd2633700 1 -- 172.21.15.89:0/3627129494 wait complete. 2023-11-24T06:20:54.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:20:54.208 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.331+0000 7f22fd65b700 1 Processor -- start 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.331+0000 7f22fd65b700 1 -- start start 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.332+0000 7f22fd65b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f8168d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.332+0000 7f22fd65b700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f80b6fb0 con 0x7f22f80b93c0 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.332+0000 7f22fd65b700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f80d2c10 con 0x7f22f815e320 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.332+0000 7f22fd65b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80d4da0 con 0x7f22f806b700 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.332+0000 7f22f67fc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f22f80b93c0 0x7f22f806aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42324/0 (socket says 172.21.15.89:42324) 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.332+0000 7f22f67fc700 1 -- 172.21.15.89:0/283263561 learned_addr learned my addr 172.21.15.89:0/283263561 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f77fe700 1 --2- 172.21.15.89:0/283263561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f8168d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f5ffb700 1 -- 172.21.15.89:0/283263561 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88527985 0 0) 0x7f22f80b6fb0 con 0x7f22f80b93c0 2023-11-24T06:20:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f5ffb700 1 -- 172.21.15.89:0/283263561 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e00035b0 con 0x7f22f80b93c0 2023-11-24T06:20:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f5ffb700 1 -- 172.21.15.89:0/283263561 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954818169 0 0) 0x7f22f80d2c10 con 0x7f22f815e320 2023-11-24T06:20:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f5ffb700 1 -- 172.21.15.89:0/283263561 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f80b6fb0 con 0x7f22f815e320 2023-11-24T06:20:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f5ffb700 1 -- 172.21.15.89:0/283263561 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4190270525 0 0) 0x7f22e00035b0 con 0x7f22f80b93c0 2023-11-24T06:20:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f77fe700 1 -- 172.21.15.89:0/283263561 >> v1:172.21.15.123:6789/0 conn(0x7f22f815e320 legacy=0x7f22f80d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f77fe700 1 -- 172.21.15.89:0/283263561 >> v1:172.21.15.89:6789/0 conn(0x7f22f80b93c0 legacy=0x7f22f806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f77fe700 1 -- 172.21.15.89:0/283263561 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f80d2ef0 con 0x7f22f806b700 2023-11-24T06:20:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f77fe700 1 --2- 172.21.15.89:0/283263561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f8168d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009dc0 tx=0x7f22f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24482c246b73d6eb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f5ffb700 1 -- 172.21.15.89:0/283263561 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f000e070 con 0x7f22f806b700 2023-11-24T06:20:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f5ffb700 1 -- 172.21.15.89:0/283263561 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f0003a20 con 0x7f22f806b700 2023-11-24T06:20:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.333+0000 7f22f5ffb700 1 -- 172.21.15.89:0/283263561 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8001670 con 0x7f22f806b700 2023-11-24T06:20:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.334+0000 7f22fd65b700 1 -- 172.21.15.89:0/283263561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 msgr2=0x7f22f8168d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.334+0000 7f22fd65b700 1 --2- 172.21.15.89:0/283263561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f8168d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009dc0 tx=0x7f22f000b330 comp rx=0 tx=0).stop 2023-11-24T06:20:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.334+0000 7f22fd65b700 1 -- 172.21.15.89:0/283263561 shutdown_connections 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.334+0000 7f22fd65b700 1 --2- 172.21.15.89:0/283263561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f8168d80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.334+0000 7f22fd65b700 1 -- 172.21.15.89:0/283263561 >> 172.21.15.89:0/283263561 conn(0x7f22f80d3e50 msgr2=0x7f22f80d4260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.334+0000 7f22fd65b700 1 -- 172.21.15.89:0/283263561 shutdown_connections 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.334+0000 7f22fd65b700 1 -- 172.21.15.89:0/283263561 wait complete. 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.335+0000 7f22fd65b700 1 Processor -- start 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.335+0000 7f22fd65b700 1 -- start start 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22fd65b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f815d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22fd65b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f80b93c0 0x7f22f80ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22fd65b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f815e320 0x7f22f80ab580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22fd65b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80cca20 con 0x7f22f80b93c0 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22fd65b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80cbe10 con 0x7f22f815e320 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22fd65b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80cc0f0 con 0x7f22f806b700 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f80b93c0 0x7f22f80ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f80b93c0 0x7f22f80ab040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38828/0 (socket says 172.21.15.89:38828) 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f67fc700 1 -- 172.21.15.89:0/3788820017 learned_addr learned my addr 172.21.15.89:0/3788820017 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f77fe700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f815e320 0x7f22f80ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f6ffd700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f815d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f67fc700 1 -- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 msgr2=0x7f22f815d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f67fc700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f815d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f67fc700 1 -- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f815e320 msgr2=0x7f22f80ab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f67fc700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f815e320 0x7f22f80ab580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f67fc700 1 -- 172.21.15.89:0/3788820017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f0009a70 con 0x7f22f80b93c0 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.336+0000 7f22f6ffd700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f815d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.337+0000 7f22f67fc700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f80b93c0 0x7f22f80ab040 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f22e80011e0 tx=0x7f22e800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.337+0000 7f22df7fe700 1 -- 172.21.15.89:0/3788820017 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8004c80 con 0x7f22f80b93c0 2023-11-24T06:20:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.337+0000 7f22df7fe700 1 -- 172.21.15.89:0/3788820017 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e8004e00 con 0x7f22f80b93c0 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.337+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f80cc3d0 con 0x7f22f80b93c0 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.337+0000 7f22df7fe700 1 -- 172.21.15.89:0/3788820017 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8007310 con 0x7f22f80b93c0 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.337+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f815d830 con 0x7f22f80b93c0 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.338+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f80cd5a0 con 0x7f22f80b93c0 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.339+0000 7f22df7fe700 1 -- 172.21.15.89:0/3788820017 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f22e8007470 con 0x7f22f80b93c0 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.339+0000 7f22df7fe700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22e406af60 0x7f22e406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.339+0000 7f22df7fe700 1 -- 172.21.15.89:0/3788820017 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f22e8089570 con 0x7f22f80b93c0 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.339+0000 7f22f6ffd700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22e406af60 0x7f22e406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.339+0000 7f22f6ffd700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22e406af60 0x7f22e406d420 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f22ec007fd0 tx=0x7f22ec009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.343+0000 7f22df7fe700 1 -- 172.21.15.89:0/3788820017 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22e8054a00 con 0x7f22f80b93c0 2023-11-24T06:20:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.535+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22f80cd5a0 con 0x7f22f80b93c0 2023-11-24T06:20:54.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.536+0000 7f22df7fe700 1 -- 172.21.15.89:0/3788820017 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22f80cd5a0 con 0x7f22f80b93c0 2023-11-24T06:20:54.538 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22e406af60 msgr2=0x7f22e406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22e406af60 0x7f22e406d420 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f22ec007fd0 tx=0x7f22ec009040 comp rx=0 tx=0).stop 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f80b93c0 msgr2=0x7f22f80ab040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f80b93c0 0x7f22f80ab040 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f22e80011e0 tx=0x7f22e800cae0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 shutdown_connections 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f22e406af60 0x7f22e406d420 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f815e320 0x7f22f80ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f806b700 0x7f22f815d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 --2- 172.21.15.89:0/3788820017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22f80b93c0 0x7f22f80ab040 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 >> 172.21.15.89:0/3788820017 conn(0x7f22f80d3e50 msgr2=0x7f22f80b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 shutdown_connections 2023-11-24T06:20:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.538+0000 7f22fd65b700 1 -- 172.21.15.89:0/3788820017 wait complete. 2023-11-24T06:20:54.552 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T06:20:54.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.671+0000 7f7f4faf7700 1 Processor -- start 2023-11-24T06:20:54.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.672+0000 7f7f4faf7700 1 -- start start 2023-11-24T06:20:54.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.672+0000 7f7f4faf7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f480b0ba0 0x7f7f480af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.672+0000 7f7f4faf7700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7f480b0f80 0x7f7f480cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.672+0000 7f7f4faf7700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f480b2bd0 con 0x7f7f4815ee90 2023-11-24T06:20:54.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.672+0000 7f7f4faf7700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f480b1fc0 con 0x7f7f480b0f80 2023-11-24T06:20:54.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.672+0000 7f7f4faf7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480b22a0 con 0x7f7f480b0ba0 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4e094700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7f4815ee90 0x7f7f480ccca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42344/0 (socket says 172.21.15.89:42344) 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4e094700 1 -- 172.21.15.89:0/2467096885 learned_addr learned my addr 172.21.15.89:0/2467096885 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4d893700 1 --2- 172.21.15.89:0/2467096885 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f480b0ba0 0x7f7f480af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4d092700 1 --2- 172.21.15.89:0/2467096885 >> v2:172.21.15.123:3300/0 conn(0x7f7f480b0f80 0x7f7f480cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4c891700 1 -- 172.21.15.89:0/2467096885 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711145075 0 0) 0x7f7f480b2bd0 con 0x7f7f4815ee90 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4c891700 1 -- 172.21.15.89:0/2467096885 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f340057c0 con 0x7f7f4815ee90 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4d893700 1 -- 172.21.15.89:0/2467096885 >> v2:172.21.15.123:3300/0 conn(0x7f7f480b0f80 msgr2=0x7f7f480cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4d893700 1 --2- 172.21.15.89:0/2467096885 >> v2:172.21.15.123:3300/0 conn(0x7f7f480b0f80 0x7f7f480cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4d893700 1 -- 172.21.15.89:0/2467096885 >> v1:172.21.15.89:6789/0 conn(0x7f7f4815ee90 legacy=0x7f7f480ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4d893700 1 -- 172.21.15.89:0/2467096885 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f480b2580 con 0x7f7f480b0ba0 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4d893700 1 --2- 172.21.15.89:0/2467096885 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f480b0ba0 0x7f7f480af5c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800a9e0 tx=0x7f7f3800acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a410dc970dc3be21 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4c891700 1 -- 172.21.15.89:0/2467096885 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3800cb70 con 0x7f7f480b0ba0 2023-11-24T06:20:54.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4c891700 1 -- 172.21.15.89:0/2467096885 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3800ccd0 con 0x7f7f480b0ba0 2023-11-24T06:20:54.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.673+0000 7f7f4c891700 1 -- 172.21.15.89:0/2467096885 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38010620 con 0x7f7f480b0ba0 2023-11-24T06:20:54.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.674+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2467096885 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f480b0ba0 msgr2=0x7f7f480af5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.674+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2467096885 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f480b0ba0 0x7f7f480af5c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800a9e0 tx=0x7f7f3800acf0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.674+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2467096885 shutdown_connections 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.674+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2467096885 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f480b0ba0 0x7f7f480af5c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.674+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2467096885 >> v2:172.21.15.123:3300/0 conn(0x7f7f480b0f80 0x7f7f480cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.674+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2467096885 >> 172.21.15.89:0/2467096885 conn(0x7f7f480b57b0 msgr2=0x7f7f480b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.674+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2467096885 shutdown_connections 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.674+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2467096885 wait complete. 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.675+0000 7f7f4faf7700 1 Processor -- start 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.675+0000 7f7f4faf7700 1 -- start start 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4faf7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480b0ba0 0x7f7f48147820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4faf7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f480b0f80 0x7f7f4814b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4faf7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f4815ee90 0x7f7f4814bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4faf7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4814cea0 con 0x7f7f480b0ba0 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4faf7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4814c000 con 0x7f7f480b0f80 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4faf7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4814c290 con 0x7f7f4815ee90 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d893700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480b0ba0 0x7f7f48147820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d893700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480b0ba0 0x7f7f48147820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38842/0 (socket says 172.21.15.89:38842) 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d893700 1 -- 172.21.15.89:0/2883443488 learned_addr learned my addr 172.21.15.89:0/2883443488 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d893700 1 -- 172.21.15.89:0/2883443488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f4815ee90 msgr2=0x7f7f4814bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d893700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f4815ee90 0x7f7f4814bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d893700 1 -- 172.21.15.89:0/2883443488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f480b0f80 msgr2=0x7f7f4814b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d092700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f480b0f80 0x7f7f4814b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d893700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f480b0f80 0x7f7f4814b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.676+0000 7f7f4d893700 1 -- 172.21.15.89:0/2883443488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3800a640 con 0x7f7f480b0ba0 2023-11-24T06:20:54.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.677+0000 7f7f4d893700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480b0ba0 0x7f7f48147820 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7f38008010 tx=0x7f7f38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:54.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.677+0000 7f7f3e7fc700 1 -- 172.21.15.89:0/2883443488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3801e070 con 0x7f7f480b0ba0 2023-11-24T06:20:54.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.677+0000 7f7f3e7fc700 1 -- 172.21.15.89:0/2883443488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3800ce40 con 0x7f7f480b0ba0 2023-11-24T06:20:54.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.677+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4814c570 con 0x7f7f480b0ba0 2023-11-24T06:20:54.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.677+0000 7f7f3e7fc700 1 -- 172.21.15.89:0/2883443488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38010a10 con 0x7f7f480b0ba0 2023-11-24T06:20:54.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.677+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4814cb30 con 0x7f7f480b0ba0 2023-11-24T06:20:54.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.678+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f48067f60 con 0x7f7f480b0ba0 2023-11-24T06:20:54.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.678+0000 7f7f3e7fc700 1 -- 172.21.15.89:0/2883443488 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f7f38022070 con 0x7f7f480b0ba0 2023-11-24T06:20:54.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.678+0000 7f7f3e7fc700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f2c06af60 0x7f7f2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:54.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.679+0000 7f7f3e7fc700 1 -- 172.21.15.89:0/2883443488 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f3808a300 con 0x7f7f480b0ba0 2023-11-24T06:20:54.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.679+0000 7f7f4d092700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f2c06af60 0x7f7f2c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:54.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.679+0000 7f7f4d092700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f2c06af60 0x7f7f2c06d420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7f480af820 tx=0x7f7f44008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:54.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.683+0000 7f7f3e7fc700 1 -- 172.21.15.89:0/2883443488 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f380557b0 con 0x7f7f480b0ba0 2023-11-24T06:20:54.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.878+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7f48071f00 con 0x7f7f480b0ba0 2023-11-24T06:20:54.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.878+0000 7f7f3e7fc700 1 -- 172.21.15.89:0/2883443488 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v60) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7f38058dd0 con 0x7f7f480b0ba0 2023-11-24T06:20:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:54.881 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-11-24T06:20:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:20:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2023-11-24T06:20:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2023-11-24T06:20:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:20:54.882 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f2c06af60 msgr2=0x7f7f2c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f2c06af60 0x7f7f2c06d420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7f480af820 tx=0x7f7f44008040 comp rx=0 tx=0).stop 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480b0ba0 msgr2=0x7f7f48147820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480b0ba0 0x7f7f48147820 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7f38008010 tx=0x7f7f38004750 comp rx=0 tx=0).stop 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 shutdown_connections 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f4815ee90 0x7f7f4814bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f480b0ba0 0x7f7f48147820 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f2c06af60 0x7f7f2c06d420 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 --2- 172.21.15.89:0/2883443488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f480b0f80 0x7f7f4814b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.880+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 >> 172.21.15.89:0/2883443488 conn(0x7f7f480b57b0 msgr2=0x7f7f4806e500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.881+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 shutdown_connections 2023-11-24T06:20:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:54.881+0000 7f7f4faf7700 1 -- 172.21.15.89:0/2883443488 wait complete. 2023-11-24T06:20:54.895 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T06:20:54.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T06:20:55.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.016+0000 7efbf9bfe700 1 Processor -- start 2023-11-24T06:20:55.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.017+0000 7efbf9bfe700 1 -- start start 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf9bfe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf41603c0 0x7efbf41607a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf9bfe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efbf40d0700 0x7efbf415cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf9bfe700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf40cf470 con 0x7efbf4073f90 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf9bfe700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf40d0270 con 0x7efbf40d0700 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf9bfe700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf40cf6a0 con 0x7efbf41603c0 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf37fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf41603c0 0x7efbf41607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf2ffd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efbf40d0700 0x7efbf415cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf37fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf41603c0 0x7efbf41607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:47726/0 (socket says 172.21.15.89:47726) 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf2ffd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efbf40d0700 0x7efbf415cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:53964/0 (socket says 172.21.15.89:53964) 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf37fe700 1 -- 172.21.15.89:0/1386321949 learned_addr learned my addr 172.21.15.89:0/1386321949 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:55.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf37fe700 1 -- 172.21.15.89:0/1386321949 >> v2:172.21.15.123:3300/0 conn(0x7efbf40d0700 msgr2=0x7efbf415cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:55.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf37fe700 1 --2- 172.21.15.89:0/1386321949 >> v2:172.21.15.123:3300/0 conn(0x7efbf40d0700 0x7efbf415cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf37fe700 1 -- 172.21.15.89:0/1386321949 >> v1:172.21.15.123:6789/0 conn(0x7efbf4073f90 legacy=0x7efbf4071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.018+0000 7efbf37fe700 1 -- 172.21.15.89:0/1386321949 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf40cf940 con 0x7efbf41603c0 2023-11-24T06:20:55.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf37fe700 1 --2- 172.21.15.89:0/1386321949 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf41603c0 0x7efbf41607a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efbe40098f0 tx=0x7efbe4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0657a0d8acd863e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:55.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf27fc700 1 -- 172.21.15.89:0/1386321949 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe4011830 con 0x7efbf41603c0 2023-11-24T06:20:55.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf27fc700 1 -- 172.21.15.89:0/1386321949 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe4011990 con 0x7efbf41603c0 2023-11-24T06:20:55.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf27fc700 1 -- 172.21.15.89:0/1386321949 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe4011d20 con 0x7efbf41603c0 2023-11-24T06:20:55.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1386321949 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf41603c0 msgr2=0x7efbf41607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1386321949 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf41603c0 0x7efbf41607a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efbe40098f0 tx=0x7efbe4009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:55.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1386321949 shutdown_connections 2023-11-24T06:20:55.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1386321949 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf41603c0 0x7efbf41607a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1386321949 >> v2:172.21.15.123:3300/0 conn(0x7efbf40d0700 0x7efbf415cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1386321949 >> 172.21.15.89:0/1386321949 conn(0x7efbf40d0af0 msgr2=0x7efbf40d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1386321949 shutdown_connections 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.019+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1386321949 wait complete. 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.021+0000 7efbf9bfe700 1 Processor -- start 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.021+0000 7efbf9bfe700 1 -- start start 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf9bfe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efbf4073f90 0x7efbf4158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf9bfe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbf41603c0 0x7efbf415af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf9bfe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf40c62c0 0x7efbf4155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf9bfe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf40c8570 con 0x7efbf4073f90 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf9bfe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf415b4a0 con 0x7efbf41603c0 2023-11-24T06:20:55.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf9bfe700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf4158d70 con 0x7efbf40c62c0 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efbf4073f90 0x7efbf4158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efbf4073f90 0x7efbf4158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38856/0 (socket says 172.21.15.89:38856) 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf37fe700 1 -- 172.21.15.89:0/1301614726 learned_addr learned my addr 172.21.15.89:0/1301614726 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf2ffd700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbf41603c0 0x7efbf415af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf3fff700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf40c62c0 0x7efbf4155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf37fe700 1 -- 172.21.15.89:0/1301614726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf40c62c0 msgr2=0x7efbf4155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf37fe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf40c62c0 0x7efbf4155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf37fe700 1 -- 172.21.15.89:0/1301614726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbf41603c0 msgr2=0x7efbf415af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf37fe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbf41603c0 0x7efbf415af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.022+0000 7efbf37fe700 1 -- 172.21.15.89:0/1301614726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe4009550 con 0x7efbf4073f90 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.023+0000 7efbf37fe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efbf4073f90 0x7efbf4158790 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7efbe4009c40 tx=0x7efbe4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:55.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.023+0000 7efbdbfff700 1 -- 172.21.15.89:0/1301614726 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe4004510 con 0x7efbf4073f90 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.023+0000 7efbdbfff700 1 -- 172.21.15.89:0/1301614726 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe401e070 con 0x7efbf4073f90 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.023+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf4159050 con 0x7efbf4073f90 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.023+0000 7efbdbfff700 1 -- 172.21.15.89:0/1301614726 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe40038e0 con 0x7efbf4073f90 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.023+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf4159610 con 0x7efbf4073f90 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.024+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf4159800 con 0x7efbf4073f90 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.024+0000 7efbdbfff700 1 -- 172.21.15.89:0/1301614726 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7efbe4003a40 con 0x7efbf4073f90 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.025+0000 7efbdbfff700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efbe006af60 0x7efbe006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.025+0000 7efbdbfff700 1 -- 172.21.15.89:0/1301614726 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efbe408a260 con 0x7efbf4073f90 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.025+0000 7efbf2ffd700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efbe006af60 0x7efbe006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.025+0000 7efbf2ffd700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efbe006af60 0x7efbe006d420 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7efbf415cff0 tx=0x7efbe8009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:55.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.029+0000 7efbdbfff700 1 -- 172.21.15.89:0/1301614726 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efbf4159800 con 0x7efbf4073f90 2023-11-24T06:20:55.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.219+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7efbf4159800 con 0x7efbf4073f90 2023-11-24T06:20:55.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.424+0000 7efbdbfff700 1 -- 172.21.15.89:0/1301614726 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v61) v1 ==== 138+0+0 (secure 0 0 0) 0x7efbf4159800 con 0x7efbf4073f90 2023-11-24T06:20:55.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efbe006af60 msgr2=0x7efbe006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efbe006af60 0x7efbe006d420 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7efbf415cff0 tx=0x7efbe8009210 comp rx=0 tx=0).stop 2023-11-24T06:20:55.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efbf4073f90 msgr2=0x7efbf4158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efbf4073f90 0x7efbf4158790 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7efbe4009c40 tx=0x7efbe4004750 comp rx=0 tx=0).stop 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 shutdown_connections 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbf40c62c0 0x7efbf4155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efbf4073f90 0x7efbf4158790 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efbe006af60 0x7efbe006d420 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 --2- 172.21.15.89:0/1301614726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbf41603c0 0x7efbf415af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 >> 172.21.15.89:0/1301614726 conn(0x7efbf40d0af0 msgr2=0x7efbf4144da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 shutdown_connections 2023-11-24T06:20:55.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.428+0000 7efbf9bfe700 1 -- 172.21.15.89:0/1301614726 wait complete. 2023-11-24T06:20:55.433 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2023-11-24T06:20:55.445 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T06:20:55.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.570+0000 7f696fcc7700 1 Processor -- start 2023-11-24T06:20:55.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.570+0000 7f696fcc7700 1 -- start start 2023-11-24T06:20:55.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696fcc7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69681587f0 0x7f696806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696fcc7700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f69680b4370 0x7f696815b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696fcc7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696806b530 con 0x7f696815b370 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696fcc7700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69680ae5e0 con 0x7f69681587f0 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696fcc7700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696815be30 con 0x7f69680b4370 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696da63700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69681587f0 0x7f696806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696da63700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69681587f0 0x7f696806aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38864/0 (socket says 172.21.15.89:38864) 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696da63700 1 -- 172.21.15.89:0/2023238037 learned_addr learned my addr 172.21.15.89:0/2023238037 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696e264700 1 --2- 172.21.15.89:0/2023238037 >> v2:172.21.15.123:3300/0 conn(0x7f69680b4370 0x7f696815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696da63700 1 -- 172.21.15.89:0/2023238037 >> v2:172.21.15.123:3300/0 conn(0x7f69680b4370 msgr2=0x7f696815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696da63700 1 --2- 172.21.15.89:0/2023238037 >> v2:172.21.15.123:3300/0 conn(0x7f69680b4370 0x7f696815b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696da63700 1 -- 172.21.15.89:0/2023238037 >> v1:172.21.15.123:6789/0 conn(0x7f696815b370 legacy=0x7f69680b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.571+0000 7f696da63700 1 -- 172.21.15.89:0/2023238037 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696815c110 con 0x7f69681587f0 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696da63700 1 --2- 172.21.15.89:0/2023238037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69681587f0 0x7f696806aff0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f69580098f0 tx=0x7f6958009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa0f2040e606b4f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696ca61700 1 -- 172.21.15.89:0/2023238037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958004230 con 0x7f69681587f0 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696ca61700 1 -- 172.21.15.89:0/2023238037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958004390 con 0x7f69681587f0 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696ca61700 1 -- 172.21.15.89:0/2023238037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69580036a0 con 0x7f69681587f0 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696fcc7700 1 -- 172.21.15.89:0/2023238037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69681587f0 msgr2=0x7f696806aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2023238037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69681587f0 0x7f696806aff0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f69580098f0 tx=0x7f6958009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696fcc7700 1 -- 172.21.15.89:0/2023238037 shutdown_connections 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2023238037 >> v2:172.21.15.123:3300/0 conn(0x7f69680b4370 0x7f696815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2023238037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69681587f0 0x7f696806aff0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696fcc7700 1 -- 172.21.15.89:0/2023238037 >> 172.21.15.89:0/2023238037 conn(0x7f69680d0bf0 msgr2=0x7f69680d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:55.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696fcc7700 1 -- 172.21.15.89:0/2023238037 shutdown_connections 2023-11-24T06:20:55.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.572+0000 7f696fcc7700 1 -- 172.21.15.89:0/2023238037 wait complete. 2023-11-24T06:20:55.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.573+0000 7f696fcc7700 1 Processor -- start 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696fcc7700 1 -- start start 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696fcc7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69680b4370 0x7f696814b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696fcc7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69681587f0 0x7f696814d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696fcc7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696815b370 0x7f696814a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696fcc7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69680b1cb0 con 0x7f696815b370 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696fcc7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696814c250 con 0x7f69680b4370 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696fcc7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696814a970 con 0x7f69681587f0 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696e264700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696815b370 0x7f696814a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696e264700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696815b370 0x7f696814a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38874/0 (socket says 172.21.15.89:38874) 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.574+0000 7f696e264700 1 -- 172.21.15.89:0/2724251233 learned_addr learned my addr 172.21.15.89:0/2724251233 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696d262700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69681587f0 0x7f696814d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696da63700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69680b4370 0x7f696814b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696e264700 1 -- 172.21.15.89:0/2724251233 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69681587f0 msgr2=0x7f696814d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696e264700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69681587f0 0x7f696814d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696e264700 1 -- 172.21.15.89:0/2724251233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69680b4370 msgr2=0x7f696814b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696e264700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69680b4370 0x7f696814b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696e264700 1 -- 172.21.15.89:0/2724251233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6958009550 con 0x7f696815b370 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696e264700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696815b370 0x7f696814a430 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6960007e40 tx=0x7f696000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f695e7fc700 1 -- 172.21.15.89:0/2724251233 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000ed30 con 0x7f696815b370 2023-11-24T06:20:55.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f695e7fc700 1 -- 172.21.15.89:0/2724251233 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960014420 con 0x7f696815b370 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f695e7fc700 1 -- 172.21.15.89:0/2724251233 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69600147d0 con 0x7f696815b370 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696814b640 con 0x7f696815b370 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.575+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696814bc00 con 0x7f696815b370 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.576+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6968068fb0 con 0x7f696815b370 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.577+0000 7f695e7fc700 1 -- 172.21.15.89:0/2724251233 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f6960014930 con 0x7f696815b370 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.577+0000 7f695e7fc700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f694c06af60 0x7f694c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.577+0000 7f695e7fc700 1 -- 172.21.15.89:0/2724251233 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6960089aa0 con 0x7f696815b370 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.577+0000 7f696da63700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f694c06af60 0x7f694c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:55.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.578+0000 7f696da63700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f694c06af60 0x7f694c06d420 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6958009130 tx=0x7f6958008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:55.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.582+0000 7f695e7fc700 1 -- 172.21.15.89:0/2724251233 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6960054d40 con 0x7f696815b370 2023-11-24T06:20:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:55.772+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f69680b4750 con 0x7f696815b370 2023-11-24T06:20:56.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.432+0000 7f695e7fc700 1 -- 172.21.15.89:0/2724251233 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v62) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6960058360 con 0x7f696815b370 2023-11-24T06:20:56.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f694c06af60 msgr2=0x7f694c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:56.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f694c06af60 0x7f694c06d420 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6958009130 tx=0x7f6958008030 comp rx=0 tx=0).stop 2023-11-24T06:20:56.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696815b370 msgr2=0x7f696814a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:56.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696815b370 0x7f696814a430 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6960007e40 tx=0x7f696000e3c0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 shutdown_connections 2023-11-24T06:20:56.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f696815b370 0x7f696814a430 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f694c06af60 0x7f694c06d420 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69680b4370 0x7f696814b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 --2- 172.21.15.89:0/2724251233 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69681587f0 0x7f696814d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 >> 172.21.15.89:0/2724251233 conn(0x7f69680d0bf0 msgr2=0x7f696806ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:56.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 shutdown_connections 2023-11-24T06:20:56.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.435+0000 7f696fcc7700 1 -- 172.21.15.89:0/2724251233 wait complete. 2023-11-24T06:20:56.439 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2023-11-24T06:20:56.462 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T06:20:56.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.587+0000 7fa77b24b700 1 Processor -- start 2023-11-24T06:20:56.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.587+0000 7fa77b24b700 1 -- start start 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa77b24b700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa77415bf40 0x7fa7740af790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa77b24b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7740cccd0 0x7fa7740cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa77b24b700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77406fd30 con 0x7fa77415c320 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa77b24b700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7740cd6a0 con 0x7fa7740cccd0 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa77b24b700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77406f770 con 0x7fa77415bf40 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa773fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7740cccd0 0x7fa7740cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa773fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7740cccd0 0x7fa7740cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38878/0 (socket says 172.21.15.89:38878) 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa773fff700 1 -- 172.21.15.89:0/632525540 learned_addr learned my addr 172.21.15.89:0/632525540 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:56.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa773fff700 1 -- 172.21.15.89:0/632525540 >> v2:172.21.15.123:3300/0 conn(0x7fa77415bf40 msgr2=0x7fa7740af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.588+0000 7fa773fff700 1 --2- 172.21.15.89:0/632525540 >> v2:172.21.15.123:3300/0 conn(0x7fa77415bf40 0x7fa7740af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa773fff700 1 -- 172.21.15.89:0/632525540 >> v1:172.21.15.123:6789/0 conn(0x7fa77415c320 legacy=0x7fa77415c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa773fff700 1 -- 172.21.15.89:0/632525540 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77406fa50 con 0x7fa7740cccd0 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa773fff700 1 --2- 172.21.15.89:0/632525540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7740cccd0 0x7fa7740cd160 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa77406cde0 tx=0x7fa760009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c668e22f86130c3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa7737fe700 1 -- 172.21.15.89:0/632525540 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760004230 con 0x7fa7740cccd0 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa7737fe700 1 -- 172.21.15.89:0/632525540 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760004390 con 0x7fa7740cccd0 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa7737fe700 1 -- 172.21.15.89:0/632525540 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600036a0 con 0x7fa7740cccd0 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa77b24b700 1 -- 172.21.15.89:0/632525540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7740cccd0 msgr2=0x7fa7740cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa77b24b700 1 --2- 172.21.15.89:0/632525540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7740cccd0 0x7fa7740cd160 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa77406cde0 tx=0x7fa760009bd0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa77b24b700 1 -- 172.21.15.89:0/632525540 shutdown_connections 2023-11-24T06:20:56.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa77b24b700 1 --2- 172.21.15.89:0/632525540 >> v2:172.21.15.123:3300/0 conn(0x7fa77415bf40 0x7fa7740af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa77b24b700 1 --2- 172.21.15.89:0/632525540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7740cccd0 0x7fa7740cd160 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.589+0000 7fa77b24b700 1 -- 172.21.15.89:0/632525540 >> 172.21.15.89:0/632525540 conn(0x7fa7740b2050 msgr2=0x7fa7740b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:56.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.590+0000 7fa77b24b700 1 -- 172.21.15.89:0/632525540 shutdown_connections 2023-11-24T06:20:56.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.590+0000 7fa77b24b700 1 -- 172.21.15.89:0/632525540 wait complete. 2023-11-24T06:20:56.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.591+0000 7fa77b24b700 1 Processor -- start 2023-11-24T06:20:56.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.591+0000 7fa77b24b700 1 -- start start 2023-11-24T06:20:56.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.591+0000 7fa77b24b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7740cccd0 0x7fa7741f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:56.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa77b24b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa77415bf40 0x7fa774145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:56.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa77b24b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77415c320 0x7fa774071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:56.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa77b24b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa774151eb0 con 0x7fa77415bf40 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa77b24b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7741512a0 con 0x7fa77415c320 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa77b24b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa774151580 con 0x7fa7740cccd0 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa77415bf40 0x7fa774145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa77415bf40 0x7fa774145c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38884/0 (socket says 172.21.15.89:38884) 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 -- 172.21.15.89:0/2392742487 learned_addr learned my addr 172.21.15.89:0/2392742487 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa7797e8700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77415c320 0x7fa774071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa778fe7700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7740cccd0 0x7fa7741f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 -- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7740cccd0 msgr2=0x7fa7741f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7740cccd0 0x7fa7741f59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 -- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77415c320 msgr2=0x7fa774071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77415c320 0x7fa774071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 -- 172.21.15.89:0/2392742487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa760009550 con 0x7fa77415bf40 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa778fe7700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7740cccd0 0x7fa7741f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa7797e8700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77415c320 0x7fa774071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa773fff700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa77415bf40 0x7fa774145c70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa774070e40 tx=0x7fa760004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa7717fa700 1 -- 172.21.15.89:0/2392742487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760003dc0 con 0x7fa77415bf40 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa7717fa700 1 -- 172.21.15.89:0/2392742487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76001f070 con 0x7fa77415bf40 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa774151860 con 0x7fa77415bf40 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa7717fa700 1 -- 172.21.15.89:0/2392742487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760011870 con 0x7fa77415bf40 2023-11-24T06:20:56.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.592+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7741461b0 con 0x7fa77415bf40 2023-11-24T06:20:56.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.593+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa774068d90 con 0x7fa77415bf40 2023-11-24T06:20:56.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.594+0000 7fa7717fa700 1 -- 172.21.15.89:0/2392742487 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fa760021030 con 0x7fa77415bf40 2023-11-24T06:20:56.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.594+0000 7fa7717fa700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa75806af60 0x7fa75806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:56.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.594+0000 7fa778fe7700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa75806af60 0x7fa75806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:56.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.594+0000 7fa7717fa700 1 -- 172.21.15.89:0/2392742487 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa76008ad80 con 0x7fa77415bf40 2023-11-24T06:20:56.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.595+0000 7fa778fe7700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa75806af60 0x7fa75806d420 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa77414fdf0 tx=0x7fa768008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:56.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.599+0000 7fa7717fa700 1 -- 172.21.15.89:0/2392742487 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa760056050 con 0x7fa77415bf40 2023-11-24T06:20:56.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:56.833+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa77414da40 con 0x7fa77415bf40 2023-11-24T06:20:57.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.433+0000 7fa7717fa700 1 -- 172.21.15.89:0/2392742487 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa77414da40 con 0x7fa77415bf40 2023-11-24T06:20:57.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.437+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa75806af60 msgr2=0x7fa75806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.437+0000 7fa77b24b700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa75806af60 0x7fa75806d420 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa77414fdf0 tx=0x7fa768008040 comp rx=0 tx=0).stop 2023-11-24T06:20:57.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa77415bf40 msgr2=0x7fa774145c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa77415bf40 0x7fa774145c70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa774070e40 tx=0x7fa760004750 comp rx=0 tx=0).stop 2023-11-24T06:20:57.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 shutdown_connections 2023-11-24T06:20:57.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa75806af60 0x7fa75806d420 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77415c320 0x7fa774071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7740cccd0 0x7fa7741f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 --2- 172.21.15.89:0/2392742487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa77415bf40 0x7fa774145c70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 >> 172.21.15.89:0/2392742487 conn(0x7fa7740b2050 msgr2=0x7fa7740cd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:57.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.438+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 shutdown_connections 2023-11-24T06:20:57.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.439+0000 7fa77b24b700 1 -- 172.21.15.89:0/2392742487 wait complete. 2023-11-24T06:20:57.442 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 10 and data pool 11 2023-11-24T06:20:57.463 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:57.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.583+0000 7f878766d700 1 Processor -- start 2023-11-24T06:20:57.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.583+0000 7f878766d700 1 -- start start 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f878766d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878006c710 0x7f87800b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f878766d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f87800af4d0 0x7f87800afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f878766d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87800ae430 con 0x7f878015e080 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f878766d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87800af0b0 con 0x7f87800af4d0 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f878766d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87800b05a0 con 0x7f878006c710 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f8785409700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f878015e080 0x7f878006c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42394/0 (socket says 172.21.15.89:42394) 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f8785409700 1 -- 172.21.15.89:0/1035448067 learned_addr learned my addr 172.21.15.89:0/1035448067 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f8785c0a700 1 --2- 172.21.15.89:0/1035448067 >> v2:172.21.15.123:3300/0 conn(0x7f87800af4d0 0x7f87800afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f8784c08700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878006c710 0x7f87800b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405786294 0 0) 0x7f87800ae430 con 0x7f878015e080 2023-11-24T06:20:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.584+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876c0035b0 con 0x7f878015e080 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556782813 0 0) 0x7f876c0035b0 con 0x7f878015e080 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878006c710 msgr2=0x7f87800b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878006c710 0x7f87800b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 >> v2:172.21.15.123:3300/0 conn(0x7f87800af4d0 msgr2=0x7f87800afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 --2- 172.21.15.89:0/1035448067 >> v2:172.21.15.123:3300/0 conn(0x7f87800af4d0 0x7f87800afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87800ae710 con 0x7f878015e080 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f8770001670 con 0x7f878015e080 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 >> v1:172.21.15.89:6789/0 conn(0x7f878015e080 legacy=0x7f878006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878006c710 0x7f876c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f876c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876c004d90 0x7f876c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87800af0b0 con 0x7f876c003fb0 2023-11-24T06:20:57.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87800b05a0 con 0x7f876c004d90 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876c00ba10 con 0x7f878006c710 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8784c08700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f876c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8785409700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878006c710 0x7f876c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8785c0a700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876c004d90 0x7f876c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8784c08700 1 -- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878006c710 msgr2=0x7f876c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8784c08700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f878006c710 0x7f876c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8784c08700 1 -- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876c004d90 msgr2=0x7f876c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8784c08700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876c004d90 0x7f876c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8784c08700 1 -- 172.21.15.89:0/1035448067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c0035b0 con 0x7f876c003fb0 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8785c0a700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876c004d90 0x7f876c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8784c08700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f876c004850 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f877c009b60 tx=0x7f876c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80d318eb42fee71 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.585+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c00e6c0 con 0x7f876c003fb0 2023-11-24T06:20:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c00e820 con 0x7f876c003fb0 2023-11-24T06:20:57.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f8777fff700 1 -- 172.21.15.89:0/1035448067 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c00eb90 con 0x7f876c003fb0 2023-11-24T06:20:57.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f878766d700 1 -- 172.21.15.89:0/1035448067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 msgr2=0x7f876c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f878766d700 1 --2- 172.21.15.89:0/1035448067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f876c004850 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f877c009b60 tx=0x7f876c005880 comp rx=0 tx=0).stop 2023-11-24T06:20:57.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f8785c0a700 1 -- 172.21.15.89:0/1035448067 reap_dead start 2023-11-24T06:20:57.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f878766d700 1 -- 172.21.15.89:0/1035448067 shutdown_connections 2023-11-24T06:20:57.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f878766d700 1 -- 172.21.15.89:0/1035448067 >> 172.21.15.89:0/1035448067 conn(0x7f8780071b90 msgr2=0x7f8780071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:57.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f878766d700 1 -- 172.21.15.89:0/1035448067 shutdown_connections 2023-11-24T06:20:57.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.586+0000 7f878766d700 1 -- 172.21.15.89:0/1035448067 wait complete. 2023-11-24T06:20:57.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.587+0000 7f878766d700 1 Processor -- start 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.587+0000 7f878766d700 1 -- start start 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f878766d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f8780148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f878766d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87800ce670 0x7f87801456b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f878766d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87800cfcb0 0x7f878015e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f878766d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780148b60 con 0x7f876c003fb0 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f878766d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87800cec10 con 0x7f87800cfcb0 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f878766d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87800ceef0 con 0x7f87800ce670 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785409700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f8780148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785409700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f8780148620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38902/0 (socket says 172.21.15.89:38902) 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785409700 1 -- 172.21.15.89:0/2035126272 learned_addr learned my addr 172.21.15.89:0/2035126272 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785c0a700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87800cfcb0 0x7f878015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8784c08700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87800ce670 0x7f87801456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785409700 1 -- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87800ce670 msgr2=0x7f87801456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785409700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87800ce670 0x7f87801456b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785409700 1 -- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87800cfcb0 msgr2=0x7f878015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785409700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87800cfcb0 0x7f878015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8785409700 1 -- 172.21.15.89:0/2035126272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87800cf1d0 con 0x7f876c003fb0 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.588+0000 7f8784c08700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87800ce670 0x7f87801456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.589+0000 7f8785409700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f8780148620 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f87700072c0 tx=0x7f87700075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.589+0000 7f8775ffb700 1 -- 172.21.15.89:0/2035126272 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87700077b0 con 0x7f876c003fb0 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.589+0000 7f8775ffb700 1 -- 172.21.15.89:0/2035126272 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877000f870 con 0x7f876c003fb0 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.589+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87800cf4b0 con 0x7f876c003fb0 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.589+0000 7f8775ffb700 1 -- 172.21.15.89:0/2035126272 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877000fc20 con 0x7f876c003fb0 2023-11-24T06:20:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.589+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8780145bf0 con 0x7f876c003fb0 2023-11-24T06:20:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.590+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87800cfb00 con 0x7f876c003fb0 2023-11-24T06:20:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.590+0000 7f8775ffb700 1 -- 172.21.15.89:0/2035126272 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f877000fd80 con 0x7f876c003fb0 2023-11-24T06:20:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.591+0000 7f8775ffb700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f876406af60 0x7f876406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.591+0000 7f8775ffb700 1 -- 172.21.15.89:0/2035126272 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8770089630 con 0x7f876c003fb0 2023-11-24T06:20:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.591+0000 7f8784c08700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f876406af60 0x7f876406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.591+0000 7f8784c08700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f876406af60 0x7f876406d420 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f877c0090b0 tx=0x7f877c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.595+0000 7f8775ffb700 1 -- 172.21.15.89:0/2035126272 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87700546f0 con 0x7f876c003fb0 2023-11-24T06:20:57.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.787+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87800753f0 con 0x7f876c003fb0 2023-11-24T06:20:57.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.789+0000 7f8775ffb700 1 -- 172.21.15.89:0/2035126272 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f8770057d10 con 0x7f876c003fb0 2023-11-24T06:20:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:57.791 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":63,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:56.833997+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:55.237439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:55.790741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:57.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f876406af60 msgr2=0x7f876406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f876406af60 0x7f876406d420 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f877c0090b0 tx=0x7f877c008030 comp rx=0 tx=0).stop 2023-11-24T06:20:57.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 msgr2=0x7f8780148620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f8780148620 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f87700072c0 tx=0x7f87700075d0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 shutdown_connections 2023-11-24T06:20:57.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f876406af60 0x7f876406d420 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87800cfcb0 0x7f878015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f876c003fb0 0x7f8780148620 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 --2- 172.21.15.89:0/2035126272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87800ce670 0x7f87801456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 >> 172.21.15.89:0/2035126272 conn(0x7f8780071b90 msgr2=0x7f878006bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:57.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 shutdown_connections 2023-11-24T06:20:57.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.791+0000 7f878766d700 1 -- 172.21.15.89:0/2035126272 wait complete. 2023-11-24T06:20:57.805 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:20:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.923+0000 7f1b77863700 1 Processor -- start 2023-11-24T06:20:57.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.924+0000 7f1b77863700 1 -- start start 2023-11-24T06:20:57.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b77863700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b700ae4e0 0x7f1b7006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b77863700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b7006c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b77863700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b700b46e0 con 0x7f1b70162240 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b77863700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b70166f50 con 0x7f1b700ae4e0 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b77863700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700b49c0 con 0x7f1b7006f360 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b74dfe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b700ae4e0 0x7f1b7006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b74dfe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b700ae4e0 0x7f1b7006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38912/0 (socket says 172.21.15.89:38912) 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b74dfe700 1 -- 172.21.15.89:0/1371185897 learned_addr learned my addr 172.21.15.89:0/1371185897 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b74dfe700 1 -- 172.21.15.89:0/1371185897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 msgr2=0x7f1b7006c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b75e00700 1 --2- 172.21.15.89:0/1371185897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b7006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b74dfe700 1 --2- 172.21.15.89:0/1371185897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b7006c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b74dfe700 1 -- 172.21.15.89:0/1371185897 >> v1:172.21.15.123:6789/0 conn(0x7f1b70162240 legacy=0x7f1b700afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:57.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b74dfe700 1 -- 172.21.15.89:0/1371185897 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700b4ca0 con 0x7f1b700ae4e0 2023-11-24T06:20:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b75e00700 1 --2- 172.21.15.89:0/1371185897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b7006c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b74dfe700 1 --2- 172.21.15.89:0/1371185897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b7006c000 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1b640098f0 tx=0x7f1b64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69f00e1276c6e33 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.925+0000 7f1b63fff700 1 -- 172.21.15.89:0/1371185897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64004230 con 0x7f1b700ae4e0 2023-11-24T06:20:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b63fff700 1 -- 172.21.15.89:0/1371185897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b64004390 con 0x7f1b700ae4e0 2023-11-24T06:20:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b63fff700 1 -- 172.21.15.89:0/1371185897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b640036a0 con 0x7f1b700ae4e0 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b77863700 1 -- 172.21.15.89:0/1371185897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 msgr2=0x7f1b7006c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b77863700 1 --2- 172.21.15.89:0/1371185897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b7006c000 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1b640098f0 tx=0x7f1b64009c00 comp rx=0 tx=0).stop 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b77863700 1 -- 172.21.15.89:0/1371185897 shutdown_connections 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b77863700 1 --2- 172.21.15.89:0/1371185897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b7006c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b77863700 1 --2- 172.21.15.89:0/1371185897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b7006c000 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b77863700 1 -- 172.21.15.89:0/1371185897 >> 172.21.15.89:0/1371185897 conn(0x7f1b70071b90 msgr2=0x7f1b70071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b77863700 1 -- 172.21.15.89:0/1371185897 shutdown_connections 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.926+0000 7f1b77863700 1 -- 172.21.15.89:0/1371185897 wait complete. 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.927+0000 7f1b77863700 1 Processor -- start 2023-11-24T06:20:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.927+0000 7f1b77863700 1 -- start start 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b77863700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b70164cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b77863700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b70160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b77863700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70162240 0x7f1b701530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b77863700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b70153620 con 0x7f1b700ae4e0 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b77863700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b70161a80 con 0x7f1b70162240 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b77863700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701655a0 con 0x7f1b7006f360 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b74dfe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b70160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b74dfe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b70160e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38918/0 (socket says 172.21.15.89:38918) 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b74dfe700 1 -- 172.21.15.89:0/3680666787 learned_addr learned my addr 172.21.15.89:0/3680666787 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b75e00700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70162240 0x7f1b701530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.928+0000 7f1b755ff700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b70164cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b74dfe700 1 -- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 msgr2=0x7f1b70164cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b74dfe700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b70164cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b74dfe700 1 -- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70162240 msgr2=0x7f1b701530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b74dfe700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70162240 0x7f1b701530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b74dfe700 1 -- 172.21.15.89:0/3680666787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64009550 con 0x7f1b700ae4e0 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b755ff700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b70164cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b75e00700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70162240 0x7f1b701530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b74dfe700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b70160e30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1b64004000 tx=0x7f1b64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b61ffb700 1 -- 172.21.15.89:0/3680666787 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6400b860 con 0x7f1b700ae4e0 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b61ffb700 1 -- 172.21.15.89:0/3680666787 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6400b9c0 con 0x7f1b700ae4e0 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b701614c0 con 0x7f1b700ae4e0 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b61ffb700 1 -- 172.21.15.89:0/3680666787 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6400bd70 con 0x7f1b700ae4e0 2023-11-24T06:20:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.929+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7015f960 con 0x7f1b700ae4e0 2023-11-24T06:20:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.930+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b700753f0 con 0x7f1b700ae4e0 2023-11-24T06:20:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.930+0000 7f1b61ffb700 1 -- 172.21.15.89:0/3680666787 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f1b64021070 con 0x7f1b700ae4e0 2023-11-24T06:20:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.931+0000 7f1b61ffb700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b5406af60 0x7f1b5406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.931+0000 7f1b61ffb700 1 -- 172.21.15.89:0/3680666787 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b6408add0 con 0x7f1b700ae4e0 2023-11-24T06:20:57.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.931+0000 7f1b755ff700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b5406af60 0x7f1b5406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.931+0000 7f1b755ff700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b5406af60 0x7f1b5406d420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009930 tx=0x7f1b6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:57.935+0000 7f1b61ffb700 1 -- 172.21.15.89:0/3680666787 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b64055f20 con 0x7f1b700ae4e0 2023-11-24T06:20:58.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.129+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b70067f60 con 0x7f1b700ae4e0 2023-11-24T06:20:58.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.130+0000 7f1b61ffb700 1 -- 172.21.15.89:0/3680666787 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f1b64059540 con 0x7f1b700ae4e0 2023-11-24T06:20:58.132 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:20:58.132 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":63,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:56.833997+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:55.237439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:55.790741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:20:58.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b5406af60 msgr2=0x7f1b5406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b5406af60 0x7f1b5406d420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009930 tx=0x7f1b6c008040 comp rx=0 tx=0).stop 2023-11-24T06:20:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 msgr2=0x7f1b70160e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b70160e30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1b64004000 tx=0x7f1b64004750 comp rx=0 tx=0).stop 2023-11-24T06:20:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 shutdown_connections 2023-11-24T06:20:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1b5406af60 0x7f1b5406d420 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70162240 0x7f1b701530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b7006f360 0x7f1b70164cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 --2- 172.21.15.89:0/3680666787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b700ae4e0 0x7f1b70160e30 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 >> 172.21.15.89:0/3680666787 conn(0x7f1b70071b90 msgr2=0x7f1b7006b180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 shutdown_connections 2023-11-24T06:20:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.132+0000 7f1b77863700 1 -- 172.21.15.89:0/3680666787 wait complete. 2023-11-24T06:20:58.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T06:20:58.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.268+0000 7fe5020cf700 1 Processor -- start 2023-11-24T06:20:58.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.268+0000 7fe5020cf700 1 -- start start 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe5020cf700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe4fc15be70 0x7fe4fc0cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe5020cf700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe4fc0746a0 0x7fe4fc0b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe5020cf700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc0b5ee0 con 0x7fe4fc0add40 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe5020cf700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4fc06bfc0 con 0x7fe4fc15be70 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe5020cf700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4fc06ba00 con 0x7fe4fc0746a0 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe4fb7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe4fc15be70 0x7fe4fc0cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe4faffd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe4fc0add40 0x7fe4fc073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42428/0 (socket says 172.21.15.89:42428) 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe4faffd700 1 -- 172.21.15.89:0/630039730 learned_addr learned my addr 172.21.15.89:0/630039730 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe4fb7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe4fc15be70 0x7fe4fc0cfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38924/0 (socket says 172.21.15.89:38924) 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.269+0000 7fe4fbfff700 1 --2- 172.21.15.89:0/630039730 >> v2:172.21.15.123:3300/0 conn(0x7fe4fc0746a0 0x7fe4fc0b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fb7fe700 1 -- 172.21.15.89:0/630039730 >> v2:172.21.15.123:3300/0 conn(0x7fe4fc0746a0 msgr2=0x7fe4fc0b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fb7fe700 1 --2- 172.21.15.89:0/630039730 >> v2:172.21.15.123:3300/0 conn(0x7fe4fc0746a0 0x7fe4fc0b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fb7fe700 1 -- 172.21.15.89:0/630039730 >> v1:172.21.15.89:6789/0 conn(0x7fe4fc0add40 legacy=0x7fe4fc073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fb7fe700 1 -- 172.21.15.89:0/630039730 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc06bce0 con 0x7fe4fc15be70 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fa7fc700 1 -- 172.21.15.89:0/630039730 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904554615 0 0) 0x7fe4fc0b5ee0 con 0x7fe4fc0add40 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fb7fe700 1 --2- 172.21.15.89:0/630039730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc15be70 0x7fe4fc0cfdf0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc06c490 tx=0x7fe4e4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ab38af17814d08 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fa7fc700 1 -- 172.21.15.89:0/630039730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4011830 con 0x7fe4fc15be70 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fa7fc700 1 -- 172.21.15.89:0/630039730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e4011990 con 0x7fe4fc15be70 2023-11-24T06:20:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe4fa7fc700 1 -- 172.21.15.89:0/630039730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4011d20 con 0x7fe4fc15be70 2023-11-24T06:20:58.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe5020cf700 1 -- 172.21.15.89:0/630039730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc15be70 msgr2=0x7fe4fc0cfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:58.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe5020cf700 1 --2- 172.21.15.89:0/630039730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc15be70 0x7fe4fc0cfdf0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc06c490 tx=0x7fe4e4009b20 comp rx=0 tx=0).stop 2023-11-24T06:20:58.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe5020cf700 1 -- 172.21.15.89:0/630039730 shutdown_connections 2023-11-24T06:20:58.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe5020cf700 1 --2- 172.21.15.89:0/630039730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc15be70 0x7fe4fc0cfdf0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe5020cf700 1 --2- 172.21.15.89:0/630039730 >> v2:172.21.15.123:3300/0 conn(0x7fe4fc0746a0 0x7fe4fc0b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe5020cf700 1 -- 172.21.15.89:0/630039730 >> 172.21.15.89:0/630039730 conn(0x7fe4fc0d0bf0 msgr2=0x7fe4fc0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:58.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe5020cf700 1 -- 172.21.15.89:0/630039730 shutdown_connections 2023-11-24T06:20:58.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.270+0000 7fe5020cf700 1 -- 172.21.15.89:0/630039730 wait complete. 2023-11-24T06:20:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.272+0000 7fe5020cf700 1 Processor -- start 2023-11-24T06:20:58.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.272+0000 7fe5020cf700 1 -- start start 2023-11-24T06:20:58.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe5020cf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc0746a0 0x7fe4fc15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:58.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe5020cf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc0add40 0x7fe4fc09e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:58.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe5020cf700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4fc15be70 0x7fe4fc15df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:58.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe5020cf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc15e490 con 0x7fe4fc0add40 2023-11-24T06:20:58.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe5020cf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc15e710 con 0x7fe4fc0746a0 2023-11-24T06:20:58.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe5020cf700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc15dcd0 con 0x7fe4fc15be70 2023-11-24T06:20:58.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc0add40 0x7fe4fc09e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:58.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc0add40 0x7fe4fc09e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38930/0 (socket says 172.21.15.89:38930) 2023-11-24T06:20:58.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 -- 172.21.15.89:0/3492385062 learned_addr learned my addr 172.21.15.89:0/3492385062 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:58.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4fb7fe700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc0746a0 0x7fe4fc15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:58.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4fbfff700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4fc15be70 0x7fe4fc15df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:58.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 -- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4fc15be70 msgr2=0x7fe4fc15df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:58.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4fc15be70 0x7fe4fc15df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 -- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc0746a0 msgr2=0x7fe4fc15f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:58.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc0746a0 0x7fe4fc15f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:58.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 -- 172.21.15.89:0/3492385062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e40094a0 con 0x7fe4fc0add40 2023-11-24T06:20:58.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4fbfff700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4fc15be70 0x7fe4fc15df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:20:58.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4fb7fe700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc0746a0 0x7fe4fc15f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:20:58.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4faffd700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc0add40 0x7fe4fc09e160 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0011e0 tx=0x7fe4ec00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4f3fff700 1 -- 172.21.15.89:0/3492385062 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec004c80 con 0x7fe4fc0add40 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4f3fff700 1 -- 172.21.15.89:0/3492385062 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec004e00 con 0x7fe4fc0add40 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc15cf10 con 0x7fe4fc0add40 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe4f3fff700 1 -- 172.21.15.89:0/3492385062 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec007310 con 0x7fe4fc0add40 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.273+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc15d4d0 con 0x7fe4fc0add40 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.274+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc068fb0 con 0x7fe4fc0add40 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.275+0000 7fe4f3fff700 1 -- 172.21.15.89:0/3492385062 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fe4ec007470 con 0x7fe4fc0add40 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.275+0000 7fe4f3fff700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe4dc06af60 0x7fe4dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.275+0000 7fe4f3fff700 1 -- 172.21.15.89:0/3492385062 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4ec089990 con 0x7fe4fc0add40 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.275+0000 7fe4fb7fe700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe4dc06af60 0x7fe4dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:58.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.276+0000 7fe4fb7fe700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe4dc06af60 0x7fe4dc06d420 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40090b0 tx=0x7fe4e401c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:58.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.279+0000 7fe4f3fff700 1 -- 172.21.15.89:0/3492385062 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4ec054b60 con 0x7fe4fc0add40 2023-11-24T06:20:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:58.516+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe4fc06cb30 con 0x7fe4fc0add40 2023-11-24T06:20:59.418 INFO:tasks.ceph.mds.b.smithi123.stderr:2023-11-24T06:20:59.417+0000 7f7efd0b2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:20:59.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.453+0000 7fe4f3fff700 1 -- 172.21.15.89:0/3492385062 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v40) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe4ec058180 con 0x7fe4fc0add40 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.455+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe4dc06af60 msgr2=0x7fe4dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.455+0000 7fe5020cf700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe4dc06af60 0x7fe4dc06d420 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40090b0 tx=0x7fe4e401c040 comp rx=0 tx=0).stop 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.455+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc0add40 msgr2=0x7fe4fc09e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.455+0000 7fe5020cf700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc0add40 0x7fe4fc09e160 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0011e0 tx=0x7fe4ec00cae0 comp rx=0 tx=0).stop 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.456+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 shutdown_connections 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.456+0000 7fe5020cf700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4fc15be70 0x7fe4fc15df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.456+0000 7fe5020cf700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe4dc06af60 0x7fe4dc06d420 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.456+0000 7fe5020cf700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4fc0746a0 0x7fe4fc15f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.456+0000 7fe5020cf700 1 --2- 172.21.15.89:0/3492385062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4fc0add40 0x7fe4fc09e160 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.456+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 >> 172.21.15.89:0/3492385062 conn(0x7fe4fc0d0bf0 msgr2=0x7fe4fc15b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.456+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 shutdown_connections 2023-11-24T06:20:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.456+0000 7fe5020cf700 1 -- 172.21.15.89:0/3492385062 wait complete. 2023-11-24T06:20:59.472 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T06:20:59.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T06:20:59.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.598+0000 7fae7a08b700 1 Processor -- start 2023-11-24T06:20:59.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.598+0000 7fae7a08b700 1 -- start start 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.598+0000 7fae7a08b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae7415b370 0x7fae7415b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.598+0000 7fae7a08b700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae740ae5e0 con 0x7fae741587f0 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.598+0000 7fae7a08b700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7415be30 con 0x7fae740b3c60 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.598+0000 7fae7a08b700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7415c110 con 0x7fae7415b370 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae72ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae7415b370 0x7fae7415b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae737fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fae741587f0 0x7fae7406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42440/0 (socket says 172.21.15.89:42440) 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae737fe700 1 -- 172.21.15.89:0/3495377934 learned_addr learned my addr 172.21.15.89:0/3495377934 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae72ffd700 1 -- 172.21.15.89:0/3495377934 >> v1:172.21.15.123:6789/0 conn(0x7fae740b3c60 legacy=0x7fae740af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:20:59.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae72ffd700 1 -- 172.21.15.89:0/3495377934 >> v1:172.21.15.89:6789/0 conn(0x7fae741587f0 legacy=0x7fae7406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae72ffd700 1 -- 172.21.15.89:0/3495377934 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae740ae860 con 0x7fae7415b370 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae727fc700 1 -- 172.21.15.89:0/3495377934 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214140609 0 0) 0x7fae740ae5e0 con 0x7fae741587f0 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae72ffd700 1 --2- 172.21.15.89:0/3495377934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae7415b800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fae68009870 tx=0x7fae68009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6feedc619607a41 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae727fc700 1 -- 172.21.15.89:0/3495377934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68004230 con 0x7fae7415b370 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae727fc700 1 -- 172.21.15.89:0/3495377934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae68004390 con 0x7fae7415b370 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.599+0000 7fae727fc700 1 -- 172.21.15.89:0/3495377934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae680036a0 con 0x7fae7415b370 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.600+0000 7fae7a08b700 1 -- 172.21.15.89:0/3495377934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 msgr2=0x7fae7415b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.600+0000 7fae7a08b700 1 --2- 172.21.15.89:0/3495377934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae7415b800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fae68009870 tx=0x7fae68009b80 comp rx=0 tx=0).stop 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.600+0000 7fae7a08b700 1 -- 172.21.15.89:0/3495377934 shutdown_connections 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.600+0000 7fae7a08b700 1 --2- 172.21.15.89:0/3495377934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae7415b800 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.600+0000 7fae7a08b700 1 -- 172.21.15.89:0/3495377934 >> 172.21.15.89:0/3495377934 conn(0x7fae740d0bf0 msgr2=0x7fae740d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:20:59.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.600+0000 7fae7a08b700 1 -- 172.21.15.89:0/3495377934 shutdown_connections 2023-11-24T06:20:59.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.600+0000 7fae7a08b700 1 -- 172.21.15.89:0/3495377934 wait complete. 2023-11-24T06:20:59.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.601+0000 7fae7a08b700 1 Processor -- start 2023-11-24T06:20:59.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.601+0000 7fae7a08b700 1 -- start start 2023-11-24T06:20:59.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae7a08b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae740b3c60 0x7fae740ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:59.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae7a08b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae741587f0 0x7fae7409e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae7a08b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae740d0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae7a08b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7409e820 con 0x7fae7415b370 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae7a08b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7409b5b0 con 0x7fae740b3c60 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae7a08b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7414a6d0 con 0x7fae741587f0 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae73fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae740d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae73fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae740d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38944/0 (socket says 172.21.15.89:38944) 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae73fff700 1 -- 172.21.15.89:0/2080132144 learned_addr learned my addr 172.21.15.89:0/2080132144 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae737fe700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae740b3c60 0x7fae740ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae72ffd700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae741587f0 0x7fae7409e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae73fff700 1 -- 172.21.15.89:0/2080132144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae741587f0 msgr2=0x7fae7409e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:59.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae73fff700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae741587f0 0x7fae7409e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae73fff700 1 -- 172.21.15.89:0/2080132144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae740b3c60 msgr2=0x7fae740ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae73fff700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae740b3c60 0x7fae740ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.602+0000 7fae73fff700 1 -- 172.21.15.89:0/2080132144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae680094a0 con 0x7fae7415b370 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.603+0000 7fae73fff700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae740d0060 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00c7a0 tx=0x7fae6c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.603+0000 7fae5bfff700 1 -- 172.21.15.89:0/2080132144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c003e10 con 0x7fae7415b370 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.603+0000 7fae5bfff700 1 -- 172.21.15.89:0/2080132144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c005070 con 0x7fae7415b370 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.603+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7414a9b0 con 0x7fae7415b370 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.603+0000 7fae5bfff700 1 -- 172.21.15.89:0/2080132144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c005420 con 0x7fae7415b370 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.603+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7409b890 con 0x7fae7415b370 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.604+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae74068fb0 con 0x7fae7415b370 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.604+0000 7fae5bfff700 1 -- 172.21.15.89:0/2080132144 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fae6c005580 con 0x7fae7415b370 2023-11-24T06:20:59.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.604+0000 7fae5bfff700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fae6006af60 0x7fae6006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:20:59.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.605+0000 7fae5bfff700 1 -- 172.21.15.89:0/2080132144 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae6c088d30 con 0x7fae7415b370 2023-11-24T06:20:59.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.605+0000 7fae737fe700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fae6006af60 0x7fae6006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:20:59.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.605+0000 7fae737fe700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fae6006af60 0x7fae6006d420 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fae64000f60 tx=0x7fae64007be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:20:59.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.609+0000 7fae5bfff700 1 -- 172.21.15.89:0/2080132144 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae6c053df0 con 0x7fae7415b370 2023-11-24T06:20:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:20:59.846+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fae740d1280 con 0x7fae7415b370 2023-11-24T06:21:00.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.465+0000 7fae5bfff700 1 -- 172.21.15.89:0/2080132144 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7fae6c017070 con 0x7fae7415b370 2023-11-24T06:21:00.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fae6006af60 msgr2=0x7fae6006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:00.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fae6006af60 0x7fae6006d420 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fae64000f60 tx=0x7fae64007be0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 msgr2=0x7fae740d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:00.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae740d0060 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00c7a0 tx=0x7fae6c00cb60 comp rx=0 tx=0).stop 2023-11-24T06:21:00.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 shutdown_connections 2023-11-24T06:21:00.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae7415b370 0x7fae740d0060 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fae6006af60 0x7fae6006d420 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae740b3c60 0x7fae740ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 --2- 172.21.15.89:0/2080132144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae741587f0 0x7fae7409e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 >> 172.21.15.89:0/2080132144 conn(0x7fae740d0bf0 msgr2=0x7fae740b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:00.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 shutdown_connections 2023-11-24T06:21:00.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.468+0000 7fae7a08b700 1 -- 172.21.15.89:0/2080132144 wait complete. 2023-11-24T06:21:00.485 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T06:21:00.485 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:21:00.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.608+0000 7f49689f3700 1 Processor -- start 2023-11-24T06:21:00.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.609+0000 7f49689f3700 1 -- start start 2023-11-24T06:21:00.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.609+0000 7f49689f3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f496415e080 0x7f49640b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.609+0000 7f49689f3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49640af4d0 0x7f49640afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.609+0000 7f49689f3700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49640ae430 con 0x7f496406c000 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.609+0000 7f49689f3700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49640af0b0 con 0x7f49640af4d0 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.609+0000 7f49689f3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49640b05a0 con 0x7f496415e080 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f4961d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49640af4d0 0x7f49640afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f4962d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f496406c000 0x7f4964073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42458/0 (socket says 172.21.15.89:42458) 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f4962d9d700 1 -- 172.21.15.89:0/2282173305 learned_addr learned my addr 172.21.15.89:0/2282173305 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f4961d9b700 1 -- 172.21.15.89:0/2282173305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f496415e080 msgr2=0x7f49640b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f4961d9b700 1 --2- 172.21.15.89:0/2282173305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f496415e080 0x7f49640b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f4961d9b700 1 -- 172.21.15.89:0/2282173305 >> v1:172.21.15.89:6789/0 conn(0x7f496406c000 legacy=0x7f4964073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f4961d9b700 1 -- 172.21.15.89:0/2282173305 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49640ae710 con 0x7f49640af4d0 2023-11-24T06:21:00.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f496159a700 1 -- 172.21.15.89:0/2282173305 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832857988 0 0) 0x7f49640ae430 con 0x7f496406c000 2023-11-24T06:21:00.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f4961d9b700 1 --2- 172.21.15.89:0/2282173305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49640af4d0 0x7f49640afb90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4954009870 tx=0x7f4954009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a004c998b48fcc8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:00.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f496159a700 1 -- 172.21.15.89:0/2282173305 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4954011830 con 0x7f49640af4d0 2023-11-24T06:21:00.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f496159a700 1 -- 172.21.15.89:0/2282173305 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4954011990 con 0x7f49640af4d0 2023-11-24T06:21:00.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.610+0000 7f496159a700 1 -- 172.21.15.89:0/2282173305 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4954011d20 con 0x7f49640af4d0 2023-11-24T06:21:00.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.611+0000 7f49689f3700 1 -- 172.21.15.89:0/2282173305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49640af4d0 msgr2=0x7f49640afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:00.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.611+0000 7f49689f3700 1 --2- 172.21.15.89:0/2282173305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49640af4d0 0x7f49640afb90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4954009870 tx=0x7f4954009b80 comp rx=0 tx=0).stop 2023-11-24T06:21:00.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.611+0000 7f49689f3700 1 -- 172.21.15.89:0/2282173305 shutdown_connections 2023-11-24T06:21:00.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.611+0000 7f49689f3700 1 --2- 172.21.15.89:0/2282173305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f496415e080 0x7f49640b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.611+0000 7f49689f3700 1 --2- 172.21.15.89:0/2282173305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49640af4d0 0x7f49640afb90 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.611+0000 7f49689f3700 1 -- 172.21.15.89:0/2282173305 >> 172.21.15.89:0/2282173305 conn(0x7f4964071b90 msgr2=0x7f4964071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:00.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.611+0000 7f49689f3700 1 -- 172.21.15.89:0/2282173305 shutdown_connections 2023-11-24T06:21:00.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.611+0000 7f49689f3700 1 -- 172.21.15.89:0/2282173305 wait complete. 2023-11-24T06:21:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.612+0000 7f49689f3700 1 Processor -- start 2023-11-24T06:21:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.612+0000 7f49689f3700 1 -- start start 2023-11-24T06:21:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f49689f3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496406c000 0x7f4964154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f49689f3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49640af4d0 0x7f49641480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f49689f3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f496415e080 0x7f4964148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f49689f3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4964158c60 con 0x7f496415e080 2023-11-24T06:21:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f49689f3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4964155420 con 0x7f496406c000 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f49689f3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4964157880 con 0x7f49640af4d0 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f496415e080 0x7f4964148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f496415e080 0x7f4964148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38960/0 (socket says 172.21.15.89:38960) 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 -- 172.21.15.89:0/1971858554 learned_addr learned my addr 172.21.15.89:0/1971858554 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4961d9b700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49640af4d0 0x7f49641480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 -- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49640af4d0 msgr2=0x7f49641480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49640af4d0 0x7f49641480d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 -- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496406c000 msgr2=0x7f4964154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f496259c700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496406c000 0x7f4964154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496406c000 0x7f4964154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 -- 172.21.15.89:0/1971858554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49540094a0 con 0x7f496415e080 2023-11-24T06:21:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f496259c700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496406c000 0x7f4964154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.613+0000 7f4962d9d700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f496415e080 0x7f4964148610 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f495c002230 tx=0x7f495c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.614+0000 7f4952ffd700 1 -- 172.21.15.89:0/1971858554 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c00ac20 con 0x7f496415e080 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.614+0000 7f4952ffd700 1 -- 172.21.15.89:0/1971858554 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495c00ada0 con 0x7f496415e080 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.614+0000 7f4952ffd700 1 -- 172.21.15.89:0/1971858554 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c007310 con 0x7f496415e080 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.614+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4964157b60 con 0x7f496415e080 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.614+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4964158120 con 0x7f496415e080 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.615+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49640753f0 con 0x7f496415e080 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.615+0000 7f4952ffd700 1 -- 172.21.15.89:0/1971858554 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f495c007470 con 0x7f496415e080 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.615+0000 7f4952ffd700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f494406af60 0x7f494406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.615+0000 7f4952ffd700 1 -- 172.21.15.89:0/1971858554 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f495c0890b0 con 0x7f496415e080 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.615+0000 7f496259c700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f494406af60 0x7f494406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:00.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.616+0000 7f496259c700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f494406af60 0x7f494406d420 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f495800a8f0 tx=0x7f4958008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.620+0000 7f4952ffd700 1 -- 172.21.15.89:0/1971858554 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f495c054200 con 0x7f496415e080 2023-11-24T06:21:00.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.858+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4964158dd0 con 0x7f496415e080 2023-11-24T06:21:00.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.858+0000 7f4952ffd700 1 -- 172.21.15.89:0/1971858554 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f4964158dd0 con 0x7f496415e080 2023-11-24T06:21:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:00.860 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/351443905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":351443905},{"type":"v1","addr":"172.21.15.123:6834","nonce":351443905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6021,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2890850053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2890850053},{"type":"v1","addr":"172.21.15.89:6836","nonce":2890850053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3912248946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3912248946},{"type":"v1","addr":"172.21.15.89:6837","nonce":3912248946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:56.834302+0000","modified":"2023-11-24T06:20:59.846940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6039":{"gid":6039,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2963752713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2963752713},{"type":"v1","addr":"172.21.15.123:6835","nonce":2963752713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T06:21:00.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f494406af60 msgr2=0x7f494406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f494406af60 0x7f494406d420 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f495800a8f0 tx=0x7f4958008040 comp rx=0 tx=0).stop 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f496415e080 msgr2=0x7f4964148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f496415e080 0x7f4964148610 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f495c002230 tx=0x7f495c00a060 comp rx=0 tx=0).stop 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 shutdown_connections 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f496415e080 0x7f4964148610 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f494406af60 0x7f494406d420 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496406c000 0x7f4964154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 --2- 172.21.15.89:0/1971858554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49640af4d0 0x7f49641480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 >> 172.21.15.89:0/1971858554 conn(0x7f4964071b90 msgr2=0x7f49640b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 shutdown_connections 2023-11-24T06:21:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.860+0000 7f49689f3700 1 -- 172.21.15.89:0/1971858554 wait complete. 2023-11-24T06:21:00.864 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 41 2023-11-24T06:21:00.875 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:21:00.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.993+0000 7f6e1689c700 1 Processor -- start 2023-11-24T06:21:00.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.993+0000 7f6e1689c700 1 -- start start 2023-11-24T06:21:00.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e1689c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e1015e080 0x7f6e100b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e1689c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6e100af4d0 0x7f6e100afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e1689c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e1006c000 0x7f6e1006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:00.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e1689c700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e100af0b0 con 0x7f6e1015e080 2023-11-24T06:21:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e1689c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1006c9f0 con 0x7f6e100af4d0 2023-11-24T06:21:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e1689c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100b05a0 con 0x7f6e1006c000 2023-11-24T06:21:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e1015e080 0x7f6e100b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e1015e080 0x7f6e100b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38964/0 (socket says 172.21.15.89:38964) 2023-11-24T06:21:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 -- 172.21.15.89:0/3895596747 learned_addr learned my addr 172.21.15.89:0/3895596747 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0f7fe700 1 --2- 172.21.15.89:0/3895596747 >> v2:172.21.15.123:3300/0 conn(0x7f6e100af4d0 0x7f6e100afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e14e39700 1 --2- 172.21.15.89:0/3895596747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e1006c000 0x7f6e1006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 -- 172.21.15.89:0/3895596747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e1006c000 msgr2=0x7f6e1006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 --2- 172.21.15.89:0/3895596747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e1006c000 0x7f6e1006c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 -- 172.21.15.89:0/3895596747 >> v2:172.21.15.123:3300/0 conn(0x7f6e100af4d0 msgr2=0x7f6e100afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 --2- 172.21.15.89:0/3895596747 >> v2:172.21.15.123:3300/0 conn(0x7f6e100af4d0 0x7f6e100afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 -- 172.21.15.89:0/3895596747 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100ae430 con 0x7f6e1015e080 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0ffff700 1 --2- 172.21.15.89:0/3895596747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100b5050 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6e1006e4e0 tx=0x7f6df8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259d06ed39eff163 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.994+0000 7f6e0effd700 1 -- 172.21.15.89:0/3895596747 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df800b870 con 0x7f6e1015e080 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e0effd700 1 -- 172.21.15.89:0/3895596747 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df800b9d0 con 0x7f6e1015e080 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e0effd700 1 -- 172.21.15.89:0/3895596747 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df800bd60 con 0x7f6e1015e080 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 -- 172.21.15.89:0/3895596747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 msgr2=0x7f6e100b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 --2- 172.21.15.89:0/3895596747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100b5050 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6e1006e4e0 tx=0x7f6df8009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 -- 172.21.15.89:0/3895596747 shutdown_connections 2023-11-24T06:21:01.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 --2- 172.21.15.89:0/3895596747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e1006c000 0x7f6e1006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 --2- 172.21.15.89:0/3895596747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100b5050 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 --2- 172.21.15.89:0/3895596747 >> v2:172.21.15.123:3300/0 conn(0x7f6e100af4d0 0x7f6e100afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 -- 172.21.15.89:0/3895596747 >> 172.21.15.89:0/3895596747 conn(0x7f6e10071b90 msgr2=0x7f6e10071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:01.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 -- 172.21.15.89:0/3895596747 shutdown_connections 2023-11-24T06:21:01.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.995+0000 7f6e1689c700 1 -- 172.21.15.89:0/3895596747 wait complete. 2023-11-24T06:21:01.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.996+0000 7f6e1689c700 1 Processor -- start 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.996+0000 7f6e1689c700 1 -- start start 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e1689c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1006c000 0x7f6e1014f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e1689c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e100af4d0 0x7f6e1014bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e1689c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e1689c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10152ae0 con 0x7f6e1015e080 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e1689c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1014f6f0 con 0x7f6e1006c000 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e1689c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1014c400 con 0x7f6e100af4d0 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e14e39700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e14e39700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100cdfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38974/0 (socket says 172.21.15.89:38974) 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e14e39700 1 -- 172.21.15.89:0/4063130058 learned_addr learned my addr 172.21.15.89:0/4063130058 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e0f7fe700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e100af4d0 0x7f6e1014bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.997+0000 7f6e0ffff700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1006c000 0x7f6e1014f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e14e39700 1 -- 172.21.15.89:0/4063130058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e100af4d0 msgr2=0x7f6e1014bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e14e39700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e100af4d0 0x7f6e1014bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e14e39700 1 -- 172.21.15.89:0/4063130058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1006c000 msgr2=0x7f6e1014f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e14e39700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1006c000 0x7f6e1014f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e14e39700 1 -- 172.21.15.89:0/4063130058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df8009550 con 0x7f6e1015e080 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e14e39700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100cdfc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6e08007e60 tx=0x7f6e0800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e0cff9700 1 -- 172.21.15.89:0/4063130058 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0800ed30 con 0x7f6e1015e080 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e0cff9700 1 -- 172.21.15.89:0/4063130058 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e08014420 con 0x7f6e1015e080 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e100c9ae0 con 0x7f6e1015e080 2023-11-24T06:21:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e0cff9700 1 -- 172.21.15.89:0/4063130058 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e080147d0 con 0x7f6e1015e080 2023-11-24T06:21:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.998+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e100c70d0 con 0x7f6e1015e080 2023-11-24T06:21:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.999+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e100753f0 con 0x7f6e1015e080 2023-11-24T06:21:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:00.999+0000 7f6e0cff9700 1 -- 172.21.15.89:0/4063130058 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f6e08014930 con 0x7f6e1015e080 2023-11-24T06:21:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.000+0000 7f6e0cff9700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6df006af60 0x7f6df006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.000+0000 7f6e0ffff700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6df006af60 0x7f6df006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.000+0000 7f6e0cff9700 1 -- 172.21.15.89:0/4063130058 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e08089be0 con 0x7f6e1015e080 2023-11-24T06:21:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.000+0000 7f6e0ffff700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6df006af60 0x7f6df006d420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6df8014040 tx=0x7f6df8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.004+0000 7f6e0cff9700 1 -- 172.21.15.89:0/4063130058 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e08054d50 con 0x7f6e1015e080 2023-11-24T06:21:01.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.196+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e100689a0 con 0x7f6e1015e080 2023-11-24T06:21:01.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.198+0000 7f6e0cff9700 1 -- 172.21.15.89:0/4063130058 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f6e08058370 con 0x7f6e1015e080 2023-11-24T06:21:01.200 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:01.200 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":64,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:58.434050+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:55.237439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:55.790741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:21:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.199+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6df006af60 msgr2=0x7f6df006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6df006af60 0x7f6df006d420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6df8014040 tx=0x7f6df8008030 comp rx=0 tx=0).stop 2023-11-24T06:21:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 msgr2=0x7f6e100cdfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100cdfc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6e08007e60 tx=0x7f6e0800e3c0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 shutdown_connections 2023-11-24T06:21:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e1015e080 0x7f6e100cdfc0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f6df006af60 0x7f6df006d420 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1006c000 0x7f6e1014f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 --2- 172.21.15.89:0/4063130058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e100af4d0 0x7f6e1014bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 >> 172.21.15.89:0/4063130058 conn(0x7f6e10071b90 msgr2=0x7f6e10073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 shutdown_connections 2023-11-24T06:21:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.200+0000 7f6e1689c700 1 -- 172.21.15.89:0/4063130058 wait complete. 2023-11-24T06:21:01.214 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T06:21:01.214 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:01.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.331+0000 7f2ccdd54700 1 Processor -- start 2023-11-24T06:21:01.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.331+0000 7f2ccdd54700 1 -- start start 2023-11-24T06:21:01.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.332+0000 7f2ccdd54700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2cc815bef0 0x7f2cc815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.332+0000 7f2ccdd54700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc806dd80 con 0x7f2cc80afe70 2023-11-24T06:21:01.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.332+0000 7f2ccdd54700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc815f880 con 0x7f2cc81624d0 2023-11-24T06:21:01.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.332+0000 7f2ccdd54700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc815ed90 con 0x7f2cc815bef0 2023-11-24T06:21:01.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.332+0000 7f2cc77fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2cc815bef0 0x7f2cc815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.332+0000 7f2cc6ffd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2cc80afe70 0x7f2cc80b4600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42484/0 (socket says 172.21.15.89:42484) 2023-11-24T06:21:01.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.332+0000 7f2cc6ffd700 1 -- 172.21.15.89:0/2605100158 learned_addr learned my addr 172.21.15.89:0/2605100158 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:01.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.333+0000 7f2cc77fe700 1 -- 172.21.15.89:0/2605100158 >> v1:172.21.15.123:6789/0 conn(0x7f2cc81624d0 legacy=0x7f2cc806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:01.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.333+0000 7f2cc77fe700 1 -- 172.21.15.89:0/2605100158 >> v1:172.21.15.89:6789/0 conn(0x7f2cc80afe70 legacy=0x7f2cc80b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.333+0000 7f2cc67fc700 1 -- 172.21.15.89:0/2605100158 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477930910 0 0) 0x7f2cc806dd80 con 0x7f2cc80afe70 2023-11-24T06:21:01.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.333+0000 7f2cc77fe700 1 -- 172.21.15.89:0/2605100158 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc815c2d0 con 0x7f2cc815bef0 2023-11-24T06:21:01.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.333+0000 7f2cc77fe700 1 --2- 172.21.15.89:0/2605100158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc815bef0 0x7f2cc815e850 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009840 tx=0x7f2cbc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6870d81a6f35ba89 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.333+0000 7f2cc67fc700 1 -- 172.21.15.89:0/2605100158 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc011830 con 0x7f2cc815bef0 2023-11-24T06:21:01.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.333+0000 7f2cc67fc700 1 -- 172.21.15.89:0/2605100158 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc011990 con 0x7f2cc815bef0 2023-11-24T06:21:01.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.333+0000 7f2cc67fc700 1 -- 172.21.15.89:0/2605100158 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc011d20 con 0x7f2cc815bef0 2023-11-24T06:21:01.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.334+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2605100158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc815bef0 msgr2=0x7f2cc815e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.334+0000 7f2ccdd54700 1 --2- 172.21.15.89:0/2605100158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc815bef0 0x7f2cc815e850 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009840 tx=0x7f2cbc009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:01.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.334+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2605100158 shutdown_connections 2023-11-24T06:21:01.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.334+0000 7f2ccdd54700 1 --2- 172.21.15.89:0/2605100158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc815bef0 0x7f2cc815e850 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.334+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2605100158 >> 172.21.15.89:0/2605100158 conn(0x7f2cc8071ac0 msgr2=0x7f2cc8071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:01.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.334+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2605100158 shutdown_connections 2023-11-24T06:21:01.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.334+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2605100158 wait complete. 2023-11-24T06:21:01.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.335+0000 7f2ccdd54700 1 Processor -- start 2023-11-24T06:21:01.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.335+0000 7f2ccdd54700 1 -- start start 2023-11-24T06:21:01.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2ccdd54700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc80afe70 0x7f2cc81f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2ccdd54700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cc815bef0 0x7f2cc80d0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2ccdd54700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2cc81624d0 0x7f2cc815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2ccdd54700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc81591b0 con 0x7f2cc80afe70 2023-11-24T06:21:01.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2ccdd54700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc8158c00 con 0x7f2cc815bef0 2023-11-24T06:21:01.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2ccdd54700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc8158ed0 con 0x7f2cc81624d0 2023-11-24T06:21:01.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc80afe70 0x7f2cc81f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc80afe70 0x7f2cc81f3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38984/0 (socket says 172.21.15.89:38984) 2023-11-24T06:21:01.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 -- 172.21.15.89:0/2461524823 learned_addr learned my addr 172.21.15.89:0/2461524823 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:01.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc6ffd700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cc815bef0 0x7f2cc80d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc7fff700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2cc81624d0 0x7f2cc815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 -- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2cc81624d0 msgr2=0x7f2cc815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2cc81624d0 0x7f2cc815d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 -- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cc815bef0 msgr2=0x7f2cc80d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cc815bef0 0x7f2cc80d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 -- 172.21.15.89:0/2461524823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc0094a0 con 0x7f2cc80afe70 2023-11-24T06:21:01.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc6ffd700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cc815bef0 0x7f2cc80d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:01.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc7fff700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2cc81624d0 0x7f2cc815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:01.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2cc77fe700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc80afe70 0x7f2cc81f3be0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0090b0 tx=0x7f2cbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2caffff700 1 -- 172.21.15.89:0/2461524823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc0036a0 con 0x7f2cc80afe70 2023-11-24T06:21:01.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.336+0000 7f2caffff700 1 -- 172.21.15.89:0/2461524823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc01f070 con 0x7f2cc80afe70 2023-11-24T06:21:01.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.337+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc81594d0 con 0x7f2cc80afe70 2023-11-24T06:21:01.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.337+0000 7f2caffff700 1 -- 172.21.15.89:0/2461524823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc003a00 con 0x7f2cc80afe70 2023-11-24T06:21:01.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.337+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc8159a90 con 0x7f2cc80afe70 2023-11-24T06:21:01.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.337+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc80728e0 con 0x7f2cc80afe70 2023-11-24T06:21:01.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.338+0000 7f2caffff700 1 -- 172.21.15.89:0/2461524823 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f2cbc003b60 con 0x7f2cc80afe70 2023-11-24T06:21:01.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.338+0000 7f2caffff700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2cb406af60 0x7f2cb406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.338+0000 7f2cc6ffd700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2cb406af60 0x7f2cb406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.338+0000 7f2caffff700 1 -- 172.21.15.89:0/2461524823 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cbc021070 con 0x7f2cc80afe70 2023-11-24T06:21:01.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.339+0000 7f2cc6ffd700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2cb406af60 0x7f2cb406d420 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8000f60 tx=0x7f2cb8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.343+0000 7f2caffff700 1 -- 172.21.15.89:0/2461524823 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cbc0561b0 con 0x7f2cc80afe70 2023-11-24T06:21:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.531+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2cc815f5a0 con 0x7f2cb406af60 2023-11-24T06:21:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.539+0000 7f2caffff700 1 -- 172.21.15.89:0/2461524823 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284020 (secure 0 0 0) 0x7f2cb80070d0 con 0x7f2cb406af60 2023-11-24T06:21:01.544 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2cb406af60 msgr2=0x7f2cb406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2cb406af60 0x7f2cb406d420 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8000f60 tx=0x7f2cb8009040 comp rx=0 tx=0).stop 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc80afe70 msgr2=0x7f2cc81f3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc80afe70 0x7f2cc81f3be0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0090b0 tx=0x7f2cbc004750 comp rx=0 tx=0).stop 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 shutdown_connections 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2cc81624d0 0x7f2cc815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc80afe70 0x7f2cc81f3be0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2cb406af60 0x7f2cb406d420 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 --2- 172.21.15.89:0/2461524823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cc815bef0 0x7f2cc80d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 >> 172.21.15.89:0/2461524823 conn(0x7f2cc8071ac0 msgr2=0x7f2cc80b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 shutdown_connections 2023-11-24T06:21:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.542+0000 7f2ccdd54700 1 -- 172.21.15.89:0/2461524823 wait complete. 2023-11-24T06:21:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:01.556 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2023-11-24T06:20:59.745160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223148,"kb_used_data":6964,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751572,"statfs":{"total":773094113280,"available":772865609728,"internally_reserved":0,"allocated":7131136,"data_stored":3701155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.319667"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441539+0000","last_change":"2023-11-24T06:20:56.441539+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441539+0000","last_undegraded":"2023-11-24T06:20:56.441539+0000","last_fullsized":"2023-11-24T06:20:56.441539+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449390+0000","last_change":"2023-11-24T06:20:56.449390+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449390+0000","last_undegraded":"2023-11-24T06:20:56.449390+0000","last_fullsized":"2023-11-24T06:20:56.449390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441773+0000","last_change":"2023-11-24T06:20:56.441773+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441773+0000","last_undegraded":"2023-11-24T06:20:56.441773+0000","last_fullsized":"2023-11-24T06:20:56.441773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446842+0000","last_change":"2023-11-24T06:20:56.446842+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446842+0000","last_undegraded":"2023-11-24T06:20:56.446842+0000","last_fullsized":"2023-11-24T06:20:56.446842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445464+0000","last_change":"2023-11-24T06:20:56.445464+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445464+0000","last_undegraded":"2023-11-24T06:20:56.445464+0000","last_fullsized":"2023-11-24T06:20:56.445464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.450162+0000","last_change":"2023-11-24T06:20:56.450162+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.450162+0000","last_undegraded":"2023-11-24T06:20:56.450162+0000","last_fullsized":"2023-11-24T06:20:56.450162+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446689+0000","last_change":"2023-11-24T06:20:56.446689+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446689+0000","last_undegraded":"2023-11-24T06:20:56.446689+0000","last_fullsized":"2023-11-24T06:20:56.446689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449944+0000","last_change":"2023-11-24T06:20:56.449944+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449944+0000","last_undegraded":"2023-11-24T06:20:56.449944+0000","last_fullsized":"2023-11-24T06:20:56.449944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.447505+0000","last_change":"2023-11-24T06:20:56.447505+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.447505+0000","last_undegraded":"2023-11-24T06:20:56.447505+0000","last_fullsized":"2023-11-24T06:20:56.447505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443357+0000","last_change":"2023-11-24T06:20:56.443357+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443357+0000","last_undegraded":"2023-11-24T06:20:56.443357+0000","last_fullsized":"2023-11-24T06:20:56.443357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440281+0000","last_change":"2023-11-24T06:20:56.440281+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440281+0000","last_undegraded":"2023-11-24T06:20:56.440281+0000","last_fullsized":"2023-11-24T06:20:56.440281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446432+0000","last_change":"2023-11-24T06:20:56.446432+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446432+0000","last_undegraded":"2023-11-24T06:20:56.446432+0000","last_fullsized":"2023-11-24T06:20:56.446432+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442484+0000","last_change":"2023-11-24T06:20:56.442484+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442484+0000","last_undegraded":"2023-11-24T06:20:56.442484+0000","last_fullsized":"2023-11-24T06:20:56.442484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445693+0000","last_change":"2023-11-24T06:20:56.445693+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445693+0000","last_undegraded":"2023-11-24T06:20:56.445693+0000","last_fullsized":"2023-11-24T06:20:56.445693+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443169+0000","last_change":"2023-11-24T06:20:56.443169+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443169+0000","last_undegraded":"2023-11-24T06:20:56.443169+0000","last_fullsized":"2023-11-24T06:20:56.443169+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443905+0000","last_change":"2023-11-24T06:20:56.443905+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443905+0000","last_undegraded":"2023-11-24T06:20:56.443905+0000","last_fullsized":"2023-11-24T06:20:56.443905+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446697+0000","last_change":"2023-11-24T06:20:56.446697+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446697+0000","last_undegraded":"2023-11-24T06:20:56.446697+0000","last_fullsized":"2023-11-24T06:20:56.446697+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444979+0000","last_change":"2023-11-24T06:20:56.444979+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444979+0000","last_undegraded":"2023-11-24T06:20:56.444979+0000","last_fullsized":"2023-11-24T06:20:56.444979+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.451453+0000","last_change":"2023-11-24T06:20:56.451453+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.451453+0000","last_undegraded":"2023-11-24T06:20:56.451453+0000","last_fullsized":"2023-11-24T06:20:56.451453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442861+0000","last_change":"2023-11-24T06:20:56.442861+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442861+0000","last_undegraded":"2023-11-24T06:20:56.442861+0000","last_fullsized":"2023-11-24T06:20:56.442861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443493+0000","last_change":"2023-11-24T06:20:56.443493+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443493+0000","last_undegraded":"2023-11-24T06:20:56.443493+0000","last_fullsized":"2023-11-24T06:20:56.443493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443677+0000","last_change":"2023-11-24T06:20:56.443677+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443677+0000","last_undegraded":"2023-11-24T06:20:56.443677+0000","last_fullsized":"2023-11-24T06:20:56.443677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438156+0000","last_change":"2023-11-24T06:20:56.438156+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438156+0000","last_undegraded":"2023-11-24T06:20:56.438156+0000","last_fullsized":"2023-11-24T06:20:56.438156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438903+0000","last_change":"2023-11-24T06:20:56.438903+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438903+0000","last_undegraded":"2023-11-24T06:20:56.438903+0000","last_fullsized":"2023-11-24T06:20:56.438903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443185+0000","last_change":"2023-11-24T06:20:56.443185+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443185+0000","last_undegraded":"2023-11-24T06:20:56.443185+0000","last_fullsized":"2023-11-24T06:20:56.443185+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446196+0000","last_change":"2023-11-24T06:20:56.446196+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446196+0000","last_undegraded":"2023-11-24T06:20:56.446196+0000","last_fullsized":"2023-11-24T06:20:56.446196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441252+0000","last_change":"2023-11-24T06:20:56.441252+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441252+0000","last_undegraded":"2023-11-24T06:20:56.441252+0000","last_fullsized":"2023-11-24T06:20:56.441252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438658+0000","last_change":"2023-11-24T06:20:56.438658+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438658+0000","last_undegraded":"2023-11-24T06:20:56.438658+0000","last_fullsized":"2023-11-24T06:20:56.438658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441558+0000","last_change":"2023-11-24T06:20:56.441558+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441558+0000","last_undegraded":"2023-11-24T06:20:56.441558+0000","last_fullsized":"2023-11-24T06:20:56.441558+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445126+0000","last_change":"2023-11-24T06:20:56.445126+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445126+0000","last_undegraded":"2023-11-24T06:20:56.445126+0000","last_fullsized":"2023-11-24T06:20:56.445126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446281+0000","last_change":"2023-11-24T06:20:56.446281+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446281+0000","last_undegraded":"2023-11-24T06:20:56.446281+0000","last_fullsized":"2023-11-24T06:20:56.446281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449025+0000","last_change":"2023-11-24T06:20:56.449025+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449025+0000","last_undegraded":"2023-11-24T06:20:56.449025+0000","last_fullsized":"2023-11-24T06:20:56.449025+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440692+0000","last_change":"2023-11-24T06:20:56.440692+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440692+0000","last_undegraded":"2023-11-24T06:20:56.440692+0000","last_fullsized":"2023-11-24T06:20:56.440692+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.455924+0000","last_change":"2023-11-24T06:20:56.450056+0000","last_active":"2023-11-24T06:20:56.455924+0000","last_peered":"2023-11-24T06:20:56.455924+0000","last_clean":"2023-11-24T06:20:56.455924+0000","last_became_active":"2023-11-24T06:20:56.449760+0000","last_became_peered":"2023-11-24T06:20:56.449760+0000","last_unstale":"2023-11-24T06:20:56.455924+0000","last_undegraded":"2023-11-24T06:20:56.455924+0000","last_fullsized":"2023-11-24T06:20:56.455924+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:42.412869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445988+0000","last_change":"2023-11-24T06:20:56.445988+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445988+0000","last_undegraded":"2023-11-24T06:20:56.445988+0000","last_fullsized":"2023-11-24T06:20:56.445988+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454721+0000","last_change":"2023-11-24T06:20:56.450515+0000","last_active":"2023-11-24T06:20:56.454721+0000","last_peered":"2023-11-24T06:20:56.454721+0000","last_clean":"2023-11-24T06:20:56.454721+0000","last_became_active":"2023-11-24T06:20:56.450219+0000","last_became_peered":"2023-11-24T06:20:56.450219+0000","last_unstale":"2023-11-24T06:20:56.454721+0000","last_undegraded":"2023-11-24T06:20:56.454721+0000","last_fullsized":"2023-11-24T06:20:56.454721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:40:15.263523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449285+0000","last_change":"2023-11-24T06:20:56.449285+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449285+0000","last_undegraded":"2023-11-24T06:20:56.449285+0000","last_fullsized":"2023-11-24T06:20:56.449285+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463396+0000","last_change":"2023-11-24T06:20:56.457862+0000","last_active":"2023-11-24T06:20:56.463396+0000","last_peered":"2023-11-24T06:20:56.463396+0000","last_clean":"2023-11-24T06:20:56.463396+0000","last_became_active":"2023-11-24T06:20:56.457485+0000","last_became_peered":"2023-11-24T06:20:56.457485+0000","last_unstale":"2023-11-24T06:20:56.463396+0000","last_undegraded":"2023-11-24T06:20:56.463396+0000","last_fullsized":"2023-11-24T06:20:56.463396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:44:24.187407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439756+0000","last_change":"2023-11-24T06:20:56.439756+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439756+0000","last_undegraded":"2023-11-24T06:20:56.439756+0000","last_fullsized":"2023-11-24T06:20:56.439756+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461058+0000","last_change":"2023-11-24T06:20:56.456592+0000","last_active":"2023-11-24T06:20:56.461058+0000","last_peered":"2023-11-24T06:20:56.461058+0000","last_clean":"2023-11-24T06:20:56.461058+0000","last_became_active":"2023-11-24T06:20:56.456416+0000","last_became_peered":"2023-11-24T06:20:56.456416+0000","last_unstale":"2023-11-24T06:20:56.461058+0000","last_undegraded":"2023-11-24T06:20:56.461058+0000","last_fullsized":"2023-11-24T06:20:56.461058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:19:08.950567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444793+0000","last_change":"2023-11-24T06:20:56.444793+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444793+0000","last_undegraded":"2023-11-24T06:20:56.444793+0000","last_fullsized":"2023-11-24T06:20:56.444793+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.455514+0000","last_change":"2023-11-24T06:20:56.452696+0000","last_active":"2023-11-24T06:20:56.455514+0000","last_peered":"2023-11-24T06:20:56.455514+0000","last_clean":"2023-11-24T06:20:56.455514+0000","last_became_active":"2023-11-24T06:20:56.452226+0000","last_became_peered":"2023-11-24T06:20:56.452226+0000","last_unstale":"2023-11-24T06:20:56.455514+0000","last_undegraded":"2023-11-24T06:20:56.455514+0000","last_fullsized":"2023-11-24T06:20:56.455514+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:59:01.228879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438821+0000","last_change":"2023-11-24T06:20:56.438821+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438821+0000","last_undegraded":"2023-11-24T06:20:56.438821+0000","last_fullsized":"2023-11-24T06:20:56.438821+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463517+0000","last_change":"2023-11-24T06:20:56.459294+0000","last_active":"2023-11-24T06:20:56.463517+0000","last_peered":"2023-11-24T06:20:56.463517+0000","last_clean":"2023-11-24T06:20:56.463517+0000","last_became_active":"2023-11-24T06:20:56.458979+0000","last_became_peered":"2023-11-24T06:20:56.458979+0000","last_unstale":"2023-11-24T06:20:56.463517+0000","last_undegraded":"2023-11-24T06:20:56.463517+0000","last_fullsized":"2023-11-24T06:20:56.463517+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:26.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446512+0000","last_change":"2023-11-24T06:20:56.446512+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446512+0000","last_undegraded":"2023-11-24T06:20:56.446512+0000","last_fullsized":"2023-11-24T06:20:56.446512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454765+0000","last_change":"2023-11-24T06:20:56.451454+0000","last_active":"2023-11-24T06:20:56.454765+0000","last_peered":"2023-11-24T06:20:56.454765+0000","last_clean":"2023-11-24T06:20:56.454765+0000","last_became_active":"2023-11-24T06:20:56.451064+0000","last_became_peered":"2023-11-24T06:20:56.451064+0000","last_unstale":"2023-11-24T06:20:56.454765+0000","last_undegraded":"2023-11-24T06:20:56.454765+0000","last_fullsized":"2023-11-24T06:20:56.454765+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442438+0000","last_change":"2023-11-24T06:20:56.442438+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442438+0000","last_undegraded":"2023-11-24T06:20:56.442438+0000","last_fullsized":"2023-11-24T06:20:56.442438+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454770+0000","last_change":"2023-11-24T06:20:56.449997+0000","last_active":"2023-11-24T06:20:56.454770+0000","last_peered":"2023-11-24T06:20:56.454770+0000","last_clean":"2023-11-24T06:20:56.454770+0000","last_became_active":"2023-11-24T06:20:56.449727+0000","last_became_peered":"2023-11-24T06:20:56.449727+0000","last_unstale":"2023-11-24T06:20:56.454770+0000","last_undegraded":"2023-11-24T06:20:56.454770+0000","last_fullsized":"2023-11-24T06:20:56.454770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:13.838572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445798+0000","last_change":"2023-11-24T06:20:56.445798+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445798+0000","last_undegraded":"2023-11-24T06:20:56.445798+0000","last_fullsized":"2023-11-24T06:20:56.445798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452073+0000","last_change":"2023-11-24T06:20:56.444835+0000","last_active":"2023-11-24T06:20:56.452073+0000","last_peered":"2023-11-24T06:20:56.452073+0000","last_clean":"2023-11-24T06:20:56.452073+0000","last_became_active":"2023-11-24T06:20:56.444547+0000","last_became_peered":"2023-11-24T06:20:56.444547+0000","last_unstale":"2023-11-24T06:20:56.452073+0000","last_undegraded":"2023-11-24T06:20:56.452073+0000","last_fullsized":"2023-11-24T06:20:56.452073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:12.584029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441093+0000","last_change":"2023-11-24T06:20:56.441093+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441093+0000","last_undegraded":"2023-11-24T06:20:56.441093+0000","last_fullsized":"2023-11-24T06:20:56.441093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458584+0000","last_change":"2023-11-24T06:20:56.455466+0000","last_active":"2023-11-24T06:20:56.458584+0000","last_peered":"2023-11-24T06:20:56.458584+0000","last_clean":"2023-11-24T06:20:56.458584+0000","last_became_active":"2023-11-24T06:20:56.455137+0000","last_became_peered":"2023-11-24T06:20:56.455137+0000","last_unstale":"2023-11-24T06:20:56.458584+0000","last_undegraded":"2023-11-24T06:20:56.458584+0000","last_fullsized":"2023-11-24T06:20:56.458584+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:25:31.853530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446530+0000","last_change":"2023-11-24T06:20:56.446530+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446530+0000","last_undegraded":"2023-11-24T06:20:56.446530+0000","last_fullsized":"2023-11-24T06:20:56.446530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.456617+0000","last_change":"2023-11-24T06:20:56.454452+0000","last_active":"2023-11-24T06:20:56.456617+0000","last_peered":"2023-11-24T06:20:56.456617+0000","last_clean":"2023-11-24T06:20:56.456617+0000","last_became_active":"2023-11-24T06:20:56.454143+0000","last_became_peered":"2023-11-24T06:20:56.454143+0000","last_unstale":"2023-11-24T06:20:56.456617+0000","last_undegraded":"2023-11-24T06:20:56.456617+0000","last_fullsized":"2023-11-24T06:20:56.456617+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:31.618289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445684+0000","last_change":"2023-11-24T06:20:56.445684+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445684+0000","last_undegraded":"2023-11-24T06:20:56.445684+0000","last_fullsized":"2023-11-24T06:20:56.445684+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454599+0000","last_change":"2023-11-24T06:20:56.450022+0000","last_active":"2023-11-24T06:20:56.454599+0000","last_peered":"2023-11-24T06:20:56.454599+0000","last_clean":"2023-11-24T06:20:56.454599+0000","last_became_active":"2023-11-24T06:20:56.449589+0000","last_became_peered":"2023-11-24T06:20:56.449589+0000","last_unstale":"2023-11-24T06:20:56.454599+0000","last_undegraded":"2023-11-24T06:20:56.454599+0000","last_fullsized":"2023-11-24T06:20:56.454599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438775+0000","last_change":"2023-11-24T06:20:56.438775+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438775+0000","last_undegraded":"2023-11-24T06:20:56.438775+0000","last_fullsized":"2023-11-24T06:20:56.438775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452082+0000","last_change":"2023-11-24T06:20:56.448978+0000","last_active":"2023-11-24T06:20:56.452082+0000","last_peered":"2023-11-24T06:20:56.452082+0000","last_clean":"2023-11-24T06:20:56.452082+0000","last_became_active":"2023-11-24T06:20:56.448657+0000","last_became_peered":"2023-11-24T06:20:56.448657+0000","last_unstale":"2023-11-24T06:20:56.452082+0000","last_undegraded":"2023-11-24T06:20:56.452082+0000","last_fullsized":"2023-11-24T06:20:56.452082+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:12.584029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443962+0000","last_change":"2023-11-24T06:20:56.443962+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443962+0000","last_undegraded":"2023-11-24T06:20:56.443962+0000","last_fullsized":"2023-11-24T06:20:56.443962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458629+0000","last_change":"2023-11-24T06:20:56.455610+0000","last_active":"2023-11-24T06:20:56.458629+0000","last_peered":"2023-11-24T06:20:56.458629+0000","last_clean":"2023-11-24T06:20:56.458629+0000","last_became_active":"2023-11-24T06:20:56.455289+0000","last_became_peered":"2023-11-24T06:20:56.455289+0000","last_unstale":"2023-11-24T06:20:56.458629+0000","last_undegraded":"2023-11-24T06:20:56.458629+0000","last_fullsized":"2023-11-24T06:20:56.458629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:17.452542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442464+0000","last_change":"2023-11-24T06:20:56.442464+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442464+0000","last_undegraded":"2023-11-24T06:20:56.442464+0000","last_fullsized":"2023-11-24T06:20:56.442464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458511+0000","last_change":"2023-11-24T06:20:56.455905+0000","last_active":"2023-11-24T06:20:56.458511+0000","last_peered":"2023-11-24T06:20:56.458511+0000","last_clean":"2023-11-24T06:20:56.458511+0000","last_became_active":"2023-11-24T06:20:56.455373+0000","last_became_peered":"2023-11-24T06:20:56.455373+0000","last_unstale":"2023-11-24T06:20:56.458511+0000","last_undegraded":"2023-11-24T06:20:56.458511+0000","last_fullsized":"2023-11-24T06:20:56.458511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446537+0000","last_change":"2023-11-24T06:20:56.446537+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446537+0000","last_undegraded":"2023-11-24T06:20:56.446537+0000","last_fullsized":"2023-11-24T06:20:56.446537+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452266+0000","last_change":"2023-11-24T06:20:56.445784+0000","last_active":"2023-11-24T06:20:56.452266+0000","last_peered":"2023-11-24T06:20:56.452266+0000","last_clean":"2023-11-24T06:20:56.452266+0000","last_became_active":"2023-11-24T06:20:56.445468+0000","last_became_peered":"2023-11-24T06:20:56.445468+0000","last_unstale":"2023-11-24T06:20:56.452266+0000","last_undegraded":"2023-11-24T06:20:56.452266+0000","last_fullsized":"2023-11-24T06:20:56.452266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442896+0000","last_change":"2023-11-24T06:20:56.442896+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442896+0000","last_undegraded":"2023-11-24T06:20:56.442896+0000","last_fullsized":"2023-11-24T06:20:56.442896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461188+0000","last_change":"2023-11-24T06:20:56.458379+0000","last_active":"2023-11-24T06:20:56.461188+0000","last_peered":"2023-11-24T06:20:56.461188+0000","last_clean":"2023-11-24T06:20:56.461188+0000","last_became_active":"2023-11-24T06:20:56.457956+0000","last_became_peered":"2023-11-24T06:20:56.457956+0000","last_unstale":"2023-11-24T06:20:56.461188+0000","last_undegraded":"2023-11-24T06:20:56.461188+0000","last_fullsized":"2023-11-24T06:20:56.461188+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:20.310219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440565+0000","last_change":"2023-11-24T06:20:56.440565+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440565+0000","last_undegraded":"2023-11-24T06:20:56.440565+0000","last_fullsized":"2023-11-24T06:20:56.440565+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463426+0000","last_change":"2023-11-24T06:20:56.459307+0000","last_active":"2023-11-24T06:20:56.463426+0000","last_peered":"2023-11-24T06:20:56.463426+0000","last_clean":"2023-11-24T06:20:56.463426+0000","last_became_active":"2023-11-24T06:20:56.459077+0000","last_became_peered":"2023-11-24T06:20:56.459077+0000","last_unstale":"2023-11-24T06:20:56.463426+0000","last_undegraded":"2023-11-24T06:20:56.463426+0000","last_fullsized":"2023-11-24T06:20:56.463426+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:07.713106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445945+0000","last_change":"2023-11-24T06:20:56.445945+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445945+0000","last_undegraded":"2023-11-24T06:20:56.445945+0000","last_fullsized":"2023-11-24T06:20:56.445945+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461147+0000","last_change":"2023-11-24T06:20:56.457538+0000","last_active":"2023-11-24T06:20:56.461147+0000","last_peered":"2023-11-24T06:20:56.461147+0000","last_clean":"2023-11-24T06:20:56.461147+0000","last_became_active":"2023-11-24T06:20:56.457247+0000","last_became_peered":"2023-11-24T06:20:56.457247+0000","last_unstale":"2023-11-24T06:20:56.461147+0000","last_undegraded":"2023-11-24T06:20:56.461147+0000","last_fullsized":"2023-11-24T06:20:56.461147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:50.332536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445239+0000","last_change":"2023-11-24T06:20:56.445239+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445239+0000","last_undegraded":"2023-11-24T06:20:56.445239+0000","last_fullsized":"2023-11-24T06:20:56.445239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.457746+0000","last_change":"2023-11-24T06:20:56.453032+0000","last_active":"2023-11-24T06:20:56.457746+0000","last_peered":"2023-11-24T06:20:56.457746+0000","last_clean":"2023-11-24T06:20:56.457746+0000","last_became_active":"2023-11-24T06:20:56.452748+0000","last_became_peered":"2023-11-24T06:20:56.452748+0000","last_unstale":"2023-11-24T06:20:56.457746+0000","last_undegraded":"2023-11-24T06:20:56.457746+0000","last_fullsized":"2023-11-24T06:20:56.457746+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:37.569663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441010+0000","last_change":"2023-11-24T06:20:56.441010+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441010+0000","last_undegraded":"2023-11-24T06:20:56.441010+0000","last_fullsized":"2023-11-24T06:20:56.441010+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463502+0000","last_change":"2023-11-24T06:20:56.459215+0000","last_active":"2023-11-24T06:20:56.463502+0000","last_peered":"2023-11-24T06:20:56.463502+0000","last_clean":"2023-11-24T06:20:56.463502+0000","last_became_active":"2023-11-24T06:20:56.458937+0000","last_became_peered":"2023-11-24T06:20:56.458937+0000","last_unstale":"2023-11-24T06:20:56.463502+0000","last_undegraded":"2023-11-24T06:20:56.463502+0000","last_fullsized":"2023-11-24T06:20:56.463502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:03:06.709201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440845+0000","last_change":"2023-11-24T06:20:56.440845+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440845+0000","last_undegraded":"2023-11-24T06:20:56.440845+0000","last_fullsized":"2023-11-24T06:20:56.440845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458490+0000","last_change":"2023-11-24T06:20:56.455809+0000","last_active":"2023-11-24T06:20:56.458490+0000","last_peered":"2023-11-24T06:20:56.458490+0000","last_clean":"2023-11-24T06:20:56.458490+0000","last_became_active":"2023-11-24T06:20:56.455461+0000","last_became_peered":"2023-11-24T06:20:56.455461+0000","last_unstale":"2023-11-24T06:20:56.458490+0000","last_undegraded":"2023-11-24T06:20:56.458490+0000","last_fullsized":"2023-11-24T06:20:56.458490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441040+0000","last_change":"2023-11-24T06:20:56.441040+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441040+0000","last_undegraded":"2023-11-24T06:20:56.441040+0000","last_fullsized":"2023-11-24T06:20:56.441040+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454779+0000","last_change":"2023-11-24T06:20:56.450795+0000","last_active":"2023-11-24T06:20:56.454779+0000","last_peered":"2023-11-24T06:20:56.454779+0000","last_clean":"2023-11-24T06:20:56.454779+0000","last_became_active":"2023-11-24T06:20:56.450111+0000","last_became_peered":"2023-11-24T06:20:56.450111+0000","last_unstale":"2023-11-24T06:20:56.454779+0000","last_undegraded":"2023-11-24T06:20:56.454779+0000","last_fullsized":"2023-11-24T06:20:56.454779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442840+0000","last_change":"2023-11-24T06:20:56.442840+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442840+0000","last_undegraded":"2023-11-24T06:20:56.442840+0000","last_fullsized":"2023-11-24T06:20:56.442840+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461173+0000","last_change":"2023-11-24T06:20:56.458175+0000","last_active":"2023-11-24T06:20:56.461173+0000","last_peered":"2023-11-24T06:20:56.461173+0000","last_clean":"2023-11-24T06:20:56.461173+0000","last_became_active":"2023-11-24T06:20:56.457570+0000","last_became_peered":"2023-11-24T06:20:56.457570+0000","last_unstale":"2023-11-24T06:20:56.461173+0000","last_undegraded":"2023-11-24T06:20:56.461173+0000","last_fullsized":"2023-11-24T06:20:56.461173+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:05.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444484+0000","last_change":"2023-11-24T06:20:56.444484+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444484+0000","last_undegraded":"2023-11-24T06:20:56.444484+0000","last_fullsized":"2023-11-24T06:20:56.444484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458402+0000","last_change":"2023-11-24T06:20:56.454563+0000","last_active":"2023-11-24T06:20:56.458402+0000","last_peered":"2023-11-24T06:20:56.458402+0000","last_clean":"2023-11-24T06:20:56.458402+0000","last_became_active":"2023-11-24T06:20:56.454014+0000","last_became_peered":"2023-11-24T06:20:56.454014+0000","last_unstale":"2023-11-24T06:20:56.458402+0000","last_undegraded":"2023-11-24T06:20:56.458402+0000","last_fullsized":"2023-11-24T06:20:56.458402+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:22:26.548969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439927+0000","last_change":"2023-11-24T06:20:56.439927+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439927+0000","last_undegraded":"2023-11-24T06:20:56.439927+0000","last_fullsized":"2023-11-24T06:20:56.439927+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458720+0000","last_change":"2023-11-24T06:20:56.455444+0000","last_active":"2023-11-24T06:20:56.458720+0000","last_peered":"2023-11-24T06:20:56.458720+0000","last_clean":"2023-11-24T06:20:56.458720+0000","last_became_active":"2023-11-24T06:20:56.455167+0000","last_became_peered":"2023-11-24T06:20:56.455167+0000","last_unstale":"2023-11-24T06:20:56.458720+0000","last_undegraded":"2023-11-24T06:20:56.458720+0000","last_fullsized":"2023-11-24T06:20:56.458720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:59.246341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439301+0000","last_change":"2023-11-24T06:20:56.439301+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439301+0000","last_undegraded":"2023-11-24T06:20:56.439301+0000","last_fullsized":"2023-11-24T06:20:56.439301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.457669+0000","last_change":"2023-11-24T06:20:56.453405+0000","last_active":"2023-11-24T06:20:56.457669+0000","last_peered":"2023-11-24T06:20:56.457669+0000","last_clean":"2023-11-24T06:20:56.457669+0000","last_became_active":"2023-11-24T06:20:56.453077+0000","last_became_peered":"2023-11-24T06:20:56.453077+0000","last_unstale":"2023-11-24T06:20:56.457669+0000","last_undegraded":"2023-11-24T06:20:56.457669+0000","last_fullsized":"2023-11-24T06:20:56.457669+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444847+0000","last_change":"2023-11-24T06:20:56.444847+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444847+0000","last_undegraded":"2023-11-24T06:20:56.444847+0000","last_fullsized":"2023-11-24T06:20:56.444847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458271+0000","last_change":"2023-11-24T06:20:56.455600+0000","last_active":"2023-11-24T06:20:56.458271+0000","last_peered":"2023-11-24T06:20:56.458271+0000","last_clean":"2023-11-24T06:20:56.458271+0000","last_became_active":"2023-11-24T06:20:56.455299+0000","last_became_peered":"2023-11-24T06:20:56.455299+0000","last_unstale":"2023-11-24T06:20:56.458271+0000","last_undegraded":"2023-11-24T06:20:56.458271+0000","last_fullsized":"2023-11-24T06:20:56.458271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:41:00.663712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446278+0000","last_change":"2023-11-24T06:20:56.446278+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446278+0000","last_undegraded":"2023-11-24T06:20:56.446278+0000","last_fullsized":"2023-11-24T06:20:56.446278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.456003+0000","last_change":"2023-11-24T06:20:56.453053+0000","last_active":"2023-11-24T06:20:56.456003+0000","last_peered":"2023-11-24T06:20:56.456003+0000","last_clean":"2023-11-24T06:20:56.456003+0000","last_became_active":"2023-11-24T06:20:56.452778+0000","last_became_peered":"2023-11-24T06:20:56.452778+0000","last_unstale":"2023-11-24T06:20:56.456003+0000","last_undegraded":"2023-11-24T06:20:56.456003+0000","last_fullsized":"2023-11-24T06:20:56.456003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:54.169232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444952+0000","last_change":"2023-11-24T06:20:56.444952+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444952+0000","last_undegraded":"2023-11-24T06:20:56.444952+0000","last_fullsized":"2023-11-24T06:20:56.444952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452274+0000","last_change":"2023-11-24T06:20:56.445884+0000","last_active":"2023-11-24T06:20:56.452274+0000","last_peered":"2023-11-24T06:20:56.452274+0000","last_clean":"2023-11-24T06:20:56.452274+0000","last_became_active":"2023-11-24T06:20:56.445600+0000","last_became_peered":"2023-11-24T06:20:56.445600+0000","last_unstale":"2023-11-24T06:20:56.452274+0000","last_undegraded":"2023-11-24T06:20:56.452274+0000","last_fullsized":"2023-11-24T06:20:56.452274+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445794+0000","last_change":"2023-11-24T06:20:56.445794+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445794+0000","last_undegraded":"2023-11-24T06:20:56.445794+0000","last_fullsized":"2023-11-24T06:20:56.445794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461106+0000","last_change":"2023-11-24T06:20:56.456769+0000","last_active":"2023-11-24T06:20:56.461106+0000","last_peered":"2023-11-24T06:20:56.461106+0000","last_clean":"2023-11-24T06:20:56.461106+0000","last_became_active":"2023-11-24T06:20:56.456558+0000","last_became_peered":"2023-11-24T06:20:56.456558+0000","last_unstale":"2023-11-24T06:20:56.461106+0000","last_undegraded":"2023-11-24T06:20:56.461106+0000","last_fullsized":"2023-11-24T06:20:56.461106+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:47.748375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440874+0000","last_change":"2023-11-24T06:20:56.440874+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440874+0000","last_undegraded":"2023-11-24T06:20:56.440874+0000","last_fullsized":"2023-11-24T06:20:56.440874+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461112+0000","last_change":"2023-11-24T06:20:56.456842+0000","last_active":"2023-11-24T06:20:56.461112+0000","last_peered":"2023-11-24T06:20:56.461112+0000","last_clean":"2023-11-24T06:20:56.461112+0000","last_became_active":"2023-11-24T06:20:56.456506+0000","last_became_peered":"2023-11-24T06:20:56.456506+0000","last_unstale":"2023-11-24T06:20:56.461112+0000","last_undegraded":"2023-11-24T06:20:56.461112+0000","last_fullsized":"2023-11-24T06:20:56.461112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:03:05.191510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":207,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-24T06:20:55.430113+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:55.430113+0000","last_peered":"2023-11-24T06:20:55.430113+0000","last_clean":"2023-11-24T06:20:55.430113+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:55.430113+0000","last_undegraded":"2023-11-24T06:20:55.430113+0000","last_fullsized":"2023-11-24T06:20:55.430113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28340,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343500,"statfs":{"total":96636764160,"available":96607744000,"internally_reserved":0,"allocated":1335296,"data_stored":904636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59699999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.505},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.628}]}]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.308},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.018}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.625}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.439}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.5}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.60899999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.65600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":753664,"data_stored":320543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.251},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.32800000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.37},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.314}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.54000000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.70299999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.73399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28328,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343512,"statfs":{"total":96636764160,"available":96607756288,"internally_reserved":0,"allocated":1335296,"data_stored":904636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.317},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.625}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.59499999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.251}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.38200000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54100000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.624}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.26500000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.247},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32500000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.432},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.47899999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.45300000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58699999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.36899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.40300000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.5}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.505},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.68200000000000005}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.35899999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41699999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:01.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:01.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.715+0000 7f2786712700 1 Processor -- start 2023-11-24T06:21:01.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.715+0000 7f2786712700 1 -- start start 2023-11-24T06:21:01.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.715+0000 7f2786712700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f27801446a0 0x7f27800b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.715+0000 7f2786712700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27800adf00 0x7f278006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.715+0000 7f2786712700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27800af950 con 0x7f278006c540 2023-11-24T06:21:01.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.715+0000 7f2786712700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278006c9f0 con 0x7f27800adf00 2023-11-24T06:21:01.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.715+0000 7f2786712700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2780144a80 con 0x7f27801446a0 2023-11-24T06:21:01.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27800adf00 0x7f278006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27800adf00 0x7f278006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38994/0 (socket says 172.21.15.89:38994) 2023-11-24T06:21:01.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277f7fe700 1 -- 172.21.15.89:0/3150164961 learned_addr learned my addr 172.21.15.89:0/3150164961 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:01.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277ffff700 1 --2- 172.21.15.89:0/3150164961 >> v2:172.21.15.123:3300/0 conn(0x7f27801446a0 0x7f27800b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277f7fe700 1 -- 172.21.15.89:0/3150164961 >> v2:172.21.15.123:3300/0 conn(0x7f27801446a0 msgr2=0x7f27800b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277f7fe700 1 --2- 172.21.15.89:0/3150164961 >> v2:172.21.15.123:3300/0 conn(0x7f27801446a0 0x7f27800b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277f7fe700 1 -- 172.21.15.89:0/3150164961 >> v1:172.21.15.123:6789/0 conn(0x7f278006c540 legacy=0x7f278006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277f7fe700 1 -- 172.21.15.89:0/3150164961 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800afc30 con 0x7f27800adf00 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277f7fe700 1 --2- 172.21.15.89:0/3150164961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f278006c000 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f27800729a0 tx=0x7f2774009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7225116bfa1f32 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277effd700 1 -- 172.21.15.89:0/3150164961 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774011830 con 0x7f27800adf00 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277effd700 1 -- 172.21.15.89:0/3150164961 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2774011990 con 0x7f27800adf00 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.716+0000 7f277effd700 1 -- 172.21.15.89:0/3150164961 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774011d20 con 0x7f27800adf00 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.717+0000 7f2786712700 1 -- 172.21.15.89:0/3150164961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 msgr2=0x7f278006c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.717+0000 7f2786712700 1 --2- 172.21.15.89:0/3150164961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f278006c000 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f27800729a0 tx=0x7f2774009c00 comp rx=0 tx=0).stop 2023-11-24T06:21:01.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.717+0000 7f2786712700 1 -- 172.21.15.89:0/3150164961 shutdown_connections 2023-11-24T06:21:01.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.717+0000 7f2786712700 1 --2- 172.21.15.89:0/3150164961 >> v2:172.21.15.123:3300/0 conn(0x7f27801446a0 0x7f27800b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.717+0000 7f2786712700 1 --2- 172.21.15.89:0/3150164961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f278006c000 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.717+0000 7f2786712700 1 -- 172.21.15.89:0/3150164961 >> 172.21.15.89:0/3150164961 conn(0x7f2780071b90 msgr2=0x7f2780071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:01.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.717+0000 7f2786712700 1 -- 172.21.15.89:0/3150164961 shutdown_connections 2023-11-24T06:21:01.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.717+0000 7f2786712700 1 -- 172.21.15.89:0/3150164961 wait complete. 2023-11-24T06:21:01.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.718+0000 7f2786712700 1 Processor -- start 2023-11-24T06:21:01.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.719+0000 7f2786712700 1 -- start start 2023-11-24T06:21:01.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.719+0000 7f2786712700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278006c540 0x7f2780141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.719+0000 7f2786712700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f2780074200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.719+0000 7f2786712700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f27801446a0 0x7f278015d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.719+0000 7f2786712700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801f76f0 con 0x7f27800adf00 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.719+0000 7f2786712700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801f79d0 con 0x7f278006c540 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.719+0000 7f2786712700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27801f7cb0 con 0x7f27801446a0 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.719+0000 7f277f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f2780074200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f2780074200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39000/0 (socket says 172.21.15.89:39000) 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277f7fe700 1 -- 172.21.15.89:0/599291182 learned_addr learned my addr 172.21.15.89:0/599291182 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277ffff700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278006c540 0x7f2780141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f2784caf700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f27801446a0 0x7f278015d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277f7fe700 1 -- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f27801446a0 msgr2=0x7f278015d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277f7fe700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f27801446a0 0x7f278015d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277f7fe700 1 -- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278006c540 msgr2=0x7f2780141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277f7fe700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278006c540 0x7f2780141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277f7fe700 1 -- 172.21.15.89:0/599291182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2774009550 con 0x7f27800adf00 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f2784caf700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f27801446a0 0x7f278015d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277f7fe700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f2780074200 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2774009c40 tx=0x7f2774004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277cff9700 1 -- 172.21.15.89:0/599291182 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774004400 con 0x7f27800adf00 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277cff9700 1 -- 172.21.15.89:0/599291182 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277401f070 con 0x7f27800adf00 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27801f7f90 con 0x7f27800adf00 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f277cff9700 1 -- 172.21.15.89:0/599291182 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27740038a0 con 0x7f27800adf00 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.720+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27801f73c0 con 0x7f27800adf00 2023-11-24T06:21:01.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.721+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27800753f0 con 0x7f27800adf00 2023-11-24T06:21:01.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.722+0000 7f277cff9700 1 -- 172.21.15.89:0/599291182 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f2774021030 con 0x7f27800adf00 2023-11-24T06:21:01.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.722+0000 7f277cff9700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f276c06af60 0x7f276c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:01.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.722+0000 7f277cff9700 1 -- 172.21.15.89:0/599291182 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f277408a920 con 0x7f27800adf00 2023-11-24T06:21:01.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.722+0000 7f277ffff700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f276c06af60 0x7f276c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:01.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.722+0000 7f277ffff700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f276c06af60 0x7f276c06d420 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f27801f7320 tx=0x7f2770009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:01.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.726+0000 7f277cff9700 1 -- 172.21.15.89:0/599291182 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2774055b20 con 0x7f27800adf00 2023-11-24T06:21:01.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.915+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27801f7b40 con 0x7f276c06af60 2023-11-24T06:21:01.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.923+0000 7f277cff9700 1 -- 172.21.15.89:0/599291182 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284020 (secure 0 0 0) 0x7f27801f7b40 con 0x7f276c06af60 2023-11-24T06:21:01.928 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:01.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f276c06af60 msgr2=0x7f276c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f276c06af60 0x7f276c06d420 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f27801f7320 tx=0x7f2770009210 comp rx=0 tx=0).stop 2023-11-24T06:21:01.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 msgr2=0x7f2780074200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:01.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f2780074200 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2774009c40 tx=0x7f2774004750 comp rx=0 tx=0).stop 2023-11-24T06:21:01.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 shutdown_connections 2023-11-24T06:21:01.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f27801446a0 0x7f278015d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f276c06af60 0x7f276c06d420 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278006c540 0x7f2780141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 --2- 172.21.15.89:0/599291182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27800adf00 0x7f2780074200 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:01.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 >> 172.21.15.89:0/599291182 conn(0x7f2780071b90 msgr2=0x7f27800b4140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:01.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 shutdown_connections 2023-11-24T06:21:01.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:01.926+0000 7f2786712700 1 -- 172.21.15.89:0/599291182 wait complete. 2023-11-24T06:21:01.932 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:01.940 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2023-11-24T06:21:01.745460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223148,"kb_used_data":6964,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751572,"statfs":{"total":773094113280,"available":772865609728,"internally_reserved":0,"allocated":7131136,"data_stored":3701155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001006"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441539+0000","last_change":"2023-11-24T06:20:56.441539+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441539+0000","last_undegraded":"2023-11-24T06:20:56.441539+0000","last_fullsized":"2023-11-24T06:20:56.441539+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449390+0000","last_change":"2023-11-24T06:20:56.449390+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449390+0000","last_undegraded":"2023-11-24T06:20:56.449390+0000","last_fullsized":"2023-11-24T06:20:56.449390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441773+0000","last_change":"2023-11-24T06:20:56.441773+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441773+0000","last_undegraded":"2023-11-24T06:20:56.441773+0000","last_fullsized":"2023-11-24T06:20:56.441773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446842+0000","last_change":"2023-11-24T06:20:56.446842+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446842+0000","last_undegraded":"2023-11-24T06:20:56.446842+0000","last_fullsized":"2023-11-24T06:20:56.446842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445464+0000","last_change":"2023-11-24T06:20:56.445464+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445464+0000","last_undegraded":"2023-11-24T06:20:56.445464+0000","last_fullsized":"2023-11-24T06:20:56.445464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.450162+0000","last_change":"2023-11-24T06:20:56.450162+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.450162+0000","last_undegraded":"2023-11-24T06:20:56.450162+0000","last_fullsized":"2023-11-24T06:20:56.450162+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446689+0000","last_change":"2023-11-24T06:20:56.446689+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446689+0000","last_undegraded":"2023-11-24T06:20:56.446689+0000","last_fullsized":"2023-11-24T06:20:56.446689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449944+0000","last_change":"2023-11-24T06:20:56.449944+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449944+0000","last_undegraded":"2023-11-24T06:20:56.449944+0000","last_fullsized":"2023-11-24T06:20:56.449944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.447505+0000","last_change":"2023-11-24T06:20:56.447505+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.447505+0000","last_undegraded":"2023-11-24T06:20:56.447505+0000","last_fullsized":"2023-11-24T06:20:56.447505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443357+0000","last_change":"2023-11-24T06:20:56.443357+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443357+0000","last_undegraded":"2023-11-24T06:20:56.443357+0000","last_fullsized":"2023-11-24T06:20:56.443357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440281+0000","last_change":"2023-11-24T06:20:56.440281+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440281+0000","last_undegraded":"2023-11-24T06:20:56.440281+0000","last_fullsized":"2023-11-24T06:20:56.440281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446432+0000","last_change":"2023-11-24T06:20:56.446432+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446432+0000","last_undegraded":"2023-11-24T06:20:56.446432+0000","last_fullsized":"2023-11-24T06:20:56.446432+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442484+0000","last_change":"2023-11-24T06:20:56.442484+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442484+0000","last_undegraded":"2023-11-24T06:20:56.442484+0000","last_fullsized":"2023-11-24T06:20:56.442484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445693+0000","last_change":"2023-11-24T06:20:56.445693+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445693+0000","last_undegraded":"2023-11-24T06:20:56.445693+0000","last_fullsized":"2023-11-24T06:20:56.445693+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443169+0000","last_change":"2023-11-24T06:20:56.443169+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443169+0000","last_undegraded":"2023-11-24T06:20:56.443169+0000","last_fullsized":"2023-11-24T06:20:56.443169+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443905+0000","last_change":"2023-11-24T06:20:56.443905+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443905+0000","last_undegraded":"2023-11-24T06:20:56.443905+0000","last_fullsized":"2023-11-24T06:20:56.443905+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446697+0000","last_change":"2023-11-24T06:20:56.446697+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446697+0000","last_undegraded":"2023-11-24T06:20:56.446697+0000","last_fullsized":"2023-11-24T06:20:56.446697+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444979+0000","last_change":"2023-11-24T06:20:56.444979+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444979+0000","last_undegraded":"2023-11-24T06:20:56.444979+0000","last_fullsized":"2023-11-24T06:20:56.444979+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.451453+0000","last_change":"2023-11-24T06:20:56.451453+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.451453+0000","last_undegraded":"2023-11-24T06:20:56.451453+0000","last_fullsized":"2023-11-24T06:20:56.451453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442861+0000","last_change":"2023-11-24T06:20:56.442861+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442861+0000","last_undegraded":"2023-11-24T06:20:56.442861+0000","last_fullsized":"2023-11-24T06:20:56.442861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443493+0000","last_change":"2023-11-24T06:20:56.443493+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443493+0000","last_undegraded":"2023-11-24T06:20:56.443493+0000","last_fullsized":"2023-11-24T06:20:56.443493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443677+0000","last_change":"2023-11-24T06:20:56.443677+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443677+0000","last_undegraded":"2023-11-24T06:20:56.443677+0000","last_fullsized":"2023-11-24T06:20:56.443677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438156+0000","last_change":"2023-11-24T06:20:56.438156+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438156+0000","last_undegraded":"2023-11-24T06:20:56.438156+0000","last_fullsized":"2023-11-24T06:20:56.438156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438903+0000","last_change":"2023-11-24T06:20:56.438903+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438903+0000","last_undegraded":"2023-11-24T06:20:56.438903+0000","last_fullsized":"2023-11-24T06:20:56.438903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443185+0000","last_change":"2023-11-24T06:20:56.443185+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443185+0000","last_undegraded":"2023-11-24T06:20:56.443185+0000","last_fullsized":"2023-11-24T06:20:56.443185+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446196+0000","last_change":"2023-11-24T06:20:56.446196+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446196+0000","last_undegraded":"2023-11-24T06:20:56.446196+0000","last_fullsized":"2023-11-24T06:20:56.446196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441252+0000","last_change":"2023-11-24T06:20:56.441252+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441252+0000","last_undegraded":"2023-11-24T06:20:56.441252+0000","last_fullsized":"2023-11-24T06:20:56.441252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438658+0000","last_change":"2023-11-24T06:20:56.438658+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438658+0000","last_undegraded":"2023-11-24T06:20:56.438658+0000","last_fullsized":"2023-11-24T06:20:56.438658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441558+0000","last_change":"2023-11-24T06:20:56.441558+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441558+0000","last_undegraded":"2023-11-24T06:20:56.441558+0000","last_fullsized":"2023-11-24T06:20:56.441558+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445126+0000","last_change":"2023-11-24T06:20:56.445126+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445126+0000","last_undegraded":"2023-11-24T06:20:56.445126+0000","last_fullsized":"2023-11-24T06:20:56.445126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446281+0000","last_change":"2023-11-24T06:20:56.446281+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446281+0000","last_undegraded":"2023-11-24T06:20:56.446281+0000","last_fullsized":"2023-11-24T06:20:56.446281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449025+0000","last_change":"2023-11-24T06:20:56.449025+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449025+0000","last_undegraded":"2023-11-24T06:20:56.449025+0000","last_fullsized":"2023-11-24T06:20:56.449025+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440692+0000","last_change":"2023-11-24T06:20:56.440692+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440692+0000","last_undegraded":"2023-11-24T06:20:56.440692+0000","last_fullsized":"2023-11-24T06:20:56.440692+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.455924+0000","last_change":"2023-11-24T06:20:56.450056+0000","last_active":"2023-11-24T06:20:56.455924+0000","last_peered":"2023-11-24T06:20:56.455924+0000","last_clean":"2023-11-24T06:20:56.455924+0000","last_became_active":"2023-11-24T06:20:56.449760+0000","last_became_peered":"2023-11-24T06:20:56.449760+0000","last_unstale":"2023-11-24T06:20:56.455924+0000","last_undegraded":"2023-11-24T06:20:56.455924+0000","last_fullsized":"2023-11-24T06:20:56.455924+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:42.412869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445988+0000","last_change":"2023-11-24T06:20:56.445988+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445988+0000","last_undegraded":"2023-11-24T06:20:56.445988+0000","last_fullsized":"2023-11-24T06:20:56.445988+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454721+0000","last_change":"2023-11-24T06:20:56.450515+0000","last_active":"2023-11-24T06:20:56.454721+0000","last_peered":"2023-11-24T06:20:56.454721+0000","last_clean":"2023-11-24T06:20:56.454721+0000","last_became_active":"2023-11-24T06:20:56.450219+0000","last_became_peered":"2023-11-24T06:20:56.450219+0000","last_unstale":"2023-11-24T06:20:56.454721+0000","last_undegraded":"2023-11-24T06:20:56.454721+0000","last_fullsized":"2023-11-24T06:20:56.454721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:40:15.263523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449285+0000","last_change":"2023-11-24T06:20:56.449285+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449285+0000","last_undegraded":"2023-11-24T06:20:56.449285+0000","last_fullsized":"2023-11-24T06:20:56.449285+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463396+0000","last_change":"2023-11-24T06:20:56.457862+0000","last_active":"2023-11-24T06:20:56.463396+0000","last_peered":"2023-11-24T06:20:56.463396+0000","last_clean":"2023-11-24T06:20:56.463396+0000","last_became_active":"2023-11-24T06:20:56.457485+0000","last_became_peered":"2023-11-24T06:20:56.457485+0000","last_unstale":"2023-11-24T06:20:56.463396+0000","last_undegraded":"2023-11-24T06:20:56.463396+0000","last_fullsized":"2023-11-24T06:20:56.463396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:44:24.187407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439756+0000","last_change":"2023-11-24T06:20:56.439756+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439756+0000","last_undegraded":"2023-11-24T06:20:56.439756+0000","last_fullsized":"2023-11-24T06:20:56.439756+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461058+0000","last_change":"2023-11-24T06:20:56.456592+0000","last_active":"2023-11-24T06:20:56.461058+0000","last_peered":"2023-11-24T06:20:56.461058+0000","last_clean":"2023-11-24T06:20:56.461058+0000","last_became_active":"2023-11-24T06:20:56.456416+0000","last_became_peered":"2023-11-24T06:20:56.456416+0000","last_unstale":"2023-11-24T06:20:56.461058+0000","last_undegraded":"2023-11-24T06:20:56.461058+0000","last_fullsized":"2023-11-24T06:20:56.461058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:19:08.950567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444793+0000","last_change":"2023-11-24T06:20:56.444793+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444793+0000","last_undegraded":"2023-11-24T06:20:56.444793+0000","last_fullsized":"2023-11-24T06:20:56.444793+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.455514+0000","last_change":"2023-11-24T06:20:56.452696+0000","last_active":"2023-11-24T06:20:56.455514+0000","last_peered":"2023-11-24T06:20:56.455514+0000","last_clean":"2023-11-24T06:20:56.455514+0000","last_became_active":"2023-11-24T06:20:56.452226+0000","last_became_peered":"2023-11-24T06:20:56.452226+0000","last_unstale":"2023-11-24T06:20:56.455514+0000","last_undegraded":"2023-11-24T06:20:56.455514+0000","last_fullsized":"2023-11-24T06:20:56.455514+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:59:01.228879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438821+0000","last_change":"2023-11-24T06:20:56.438821+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438821+0000","last_undegraded":"2023-11-24T06:20:56.438821+0000","last_fullsized":"2023-11-24T06:20:56.438821+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463517+0000","last_change":"2023-11-24T06:20:56.459294+0000","last_active":"2023-11-24T06:20:56.463517+0000","last_peered":"2023-11-24T06:20:56.463517+0000","last_clean":"2023-11-24T06:20:56.463517+0000","last_became_active":"2023-11-24T06:20:56.458979+0000","last_became_peered":"2023-11-24T06:20:56.458979+0000","last_unstale":"2023-11-24T06:20:56.463517+0000","last_undegraded":"2023-11-24T06:20:56.463517+0000","last_fullsized":"2023-11-24T06:20:56.463517+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:26.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446512+0000","last_change":"2023-11-24T06:20:56.446512+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446512+0000","last_undegraded":"2023-11-24T06:20:56.446512+0000","last_fullsized":"2023-11-24T06:20:56.446512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454765+0000","last_change":"2023-11-24T06:20:56.451454+0000","last_active":"2023-11-24T06:20:56.454765+0000","last_peered":"2023-11-24T06:20:56.454765+0000","last_clean":"2023-11-24T06:20:56.454765+0000","last_became_active":"2023-11-24T06:20:56.451064+0000","last_became_peered":"2023-11-24T06:20:56.451064+0000","last_unstale":"2023-11-24T06:20:56.454765+0000","last_undegraded":"2023-11-24T06:20:56.454765+0000","last_fullsized":"2023-11-24T06:20:56.454765+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442438+0000","last_change":"2023-11-24T06:20:56.442438+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442438+0000","last_undegraded":"2023-11-24T06:20:56.442438+0000","last_fullsized":"2023-11-24T06:20:56.442438+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454770+0000","last_change":"2023-11-24T06:20:56.449997+0000","last_active":"2023-11-24T06:20:56.454770+0000","last_peered":"2023-11-24T06:20:56.454770+0000","last_clean":"2023-11-24T06:20:56.454770+0000","last_became_active":"2023-11-24T06:20:56.449727+0000","last_became_peered":"2023-11-24T06:20:56.449727+0000","last_unstale":"2023-11-24T06:20:56.454770+0000","last_undegraded":"2023-11-24T06:20:56.454770+0000","last_fullsized":"2023-11-24T06:20:56.454770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:13.838572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445798+0000","last_change":"2023-11-24T06:20:56.445798+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445798+0000","last_undegraded":"2023-11-24T06:20:56.445798+0000","last_fullsized":"2023-11-24T06:20:56.445798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452073+0000","last_change":"2023-11-24T06:20:56.444835+0000","last_active":"2023-11-24T06:20:56.452073+0000","last_peered":"2023-11-24T06:20:56.452073+0000","last_clean":"2023-11-24T06:20:56.452073+0000","last_became_active":"2023-11-24T06:20:56.444547+0000","last_became_peered":"2023-11-24T06:20:56.444547+0000","last_unstale":"2023-11-24T06:20:56.452073+0000","last_undegraded":"2023-11-24T06:20:56.452073+0000","last_fullsized":"2023-11-24T06:20:56.452073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:12.584029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441093+0000","last_change":"2023-11-24T06:20:56.441093+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441093+0000","last_undegraded":"2023-11-24T06:20:56.441093+0000","last_fullsized":"2023-11-24T06:20:56.441093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458584+0000","last_change":"2023-11-24T06:20:56.455466+0000","last_active":"2023-11-24T06:20:56.458584+0000","last_peered":"2023-11-24T06:20:56.458584+0000","last_clean":"2023-11-24T06:20:56.458584+0000","last_became_active":"2023-11-24T06:20:56.455137+0000","last_became_peered":"2023-11-24T06:20:56.455137+0000","last_unstale":"2023-11-24T06:20:56.458584+0000","last_undegraded":"2023-11-24T06:20:56.458584+0000","last_fullsized":"2023-11-24T06:20:56.458584+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:25:31.853530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446530+0000","last_change":"2023-11-24T06:20:56.446530+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446530+0000","last_undegraded":"2023-11-24T06:20:56.446530+0000","last_fullsized":"2023-11-24T06:20:56.446530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.456617+0000","last_change":"2023-11-24T06:20:56.454452+0000","last_active":"2023-11-24T06:20:56.456617+0000","last_peered":"2023-11-24T06:20:56.456617+0000","last_clean":"2023-11-24T06:20:56.456617+0000","last_became_active":"2023-11-24T06:20:56.454143+0000","last_became_peered":"2023-11-24T06:20:56.454143+0000","last_unstale":"2023-11-24T06:20:56.456617+0000","last_undegraded":"2023-11-24T06:20:56.456617+0000","last_fullsized":"2023-11-24T06:20:56.456617+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:31.618289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445684+0000","last_change":"2023-11-24T06:20:56.445684+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445684+0000","last_undegraded":"2023-11-24T06:20:56.445684+0000","last_fullsized":"2023-11-24T06:20:56.445684+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454599+0000","last_change":"2023-11-24T06:20:56.450022+0000","last_active":"2023-11-24T06:20:56.454599+0000","last_peered":"2023-11-24T06:20:56.454599+0000","last_clean":"2023-11-24T06:20:56.454599+0000","last_became_active":"2023-11-24T06:20:56.449589+0000","last_became_peered":"2023-11-24T06:20:56.449589+0000","last_unstale":"2023-11-24T06:20:56.454599+0000","last_undegraded":"2023-11-24T06:20:56.454599+0000","last_fullsized":"2023-11-24T06:20:56.454599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438775+0000","last_change":"2023-11-24T06:20:56.438775+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438775+0000","last_undegraded":"2023-11-24T06:20:56.438775+0000","last_fullsized":"2023-11-24T06:20:56.438775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452082+0000","last_change":"2023-11-24T06:20:56.448978+0000","last_active":"2023-11-24T06:20:56.452082+0000","last_peered":"2023-11-24T06:20:56.452082+0000","last_clean":"2023-11-24T06:20:56.452082+0000","last_became_active":"2023-11-24T06:20:56.448657+0000","last_became_peered":"2023-11-24T06:20:56.448657+0000","last_unstale":"2023-11-24T06:20:56.452082+0000","last_undegraded":"2023-11-24T06:20:56.452082+0000","last_fullsized":"2023-11-24T06:20:56.452082+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:12.584029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443962+0000","last_change":"2023-11-24T06:20:56.443962+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443962+0000","last_undegraded":"2023-11-24T06:20:56.443962+0000","last_fullsized":"2023-11-24T06:20:56.443962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458629+0000","last_change":"2023-11-24T06:20:56.455610+0000","last_active":"2023-11-24T06:20:56.458629+0000","last_peered":"2023-11-24T06:20:56.458629+0000","last_clean":"2023-11-24T06:20:56.458629+0000","last_became_active":"2023-11-24T06:20:56.455289+0000","last_became_peered":"2023-11-24T06:20:56.455289+0000","last_unstale":"2023-11-24T06:20:56.458629+0000","last_undegraded":"2023-11-24T06:20:56.458629+0000","last_fullsized":"2023-11-24T06:20:56.458629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:17.452542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442464+0000","last_change":"2023-11-24T06:20:56.442464+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442464+0000","last_undegraded":"2023-11-24T06:20:56.442464+0000","last_fullsized":"2023-11-24T06:20:56.442464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458511+0000","last_change":"2023-11-24T06:20:56.455905+0000","last_active":"2023-11-24T06:20:56.458511+0000","last_peered":"2023-11-24T06:20:56.458511+0000","last_clean":"2023-11-24T06:20:56.458511+0000","last_became_active":"2023-11-24T06:20:56.455373+0000","last_became_peered":"2023-11-24T06:20:56.455373+0000","last_unstale":"2023-11-24T06:20:56.458511+0000","last_undegraded":"2023-11-24T06:20:56.458511+0000","last_fullsized":"2023-11-24T06:20:56.458511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446537+0000","last_change":"2023-11-24T06:20:56.446537+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446537+0000","last_undegraded":"2023-11-24T06:20:56.446537+0000","last_fullsized":"2023-11-24T06:20:56.446537+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452266+0000","last_change":"2023-11-24T06:20:56.445784+0000","last_active":"2023-11-24T06:20:56.452266+0000","last_peered":"2023-11-24T06:20:56.452266+0000","last_clean":"2023-11-24T06:20:56.452266+0000","last_became_active":"2023-11-24T06:20:56.445468+0000","last_became_peered":"2023-11-24T06:20:56.445468+0000","last_unstale":"2023-11-24T06:20:56.452266+0000","last_undegraded":"2023-11-24T06:20:56.452266+0000","last_fullsized":"2023-11-24T06:20:56.452266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442896+0000","last_change":"2023-11-24T06:20:56.442896+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442896+0000","last_undegraded":"2023-11-24T06:20:56.442896+0000","last_fullsized":"2023-11-24T06:20:56.442896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461188+0000","last_change":"2023-11-24T06:20:56.458379+0000","last_active":"2023-11-24T06:20:56.461188+0000","last_peered":"2023-11-24T06:20:56.461188+0000","last_clean":"2023-11-24T06:20:56.461188+0000","last_became_active":"2023-11-24T06:20:56.457956+0000","last_became_peered":"2023-11-24T06:20:56.457956+0000","last_unstale":"2023-11-24T06:20:56.461188+0000","last_undegraded":"2023-11-24T06:20:56.461188+0000","last_fullsized":"2023-11-24T06:20:56.461188+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:20.310219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440565+0000","last_change":"2023-11-24T06:20:56.440565+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440565+0000","last_undegraded":"2023-11-24T06:20:56.440565+0000","last_fullsized":"2023-11-24T06:20:56.440565+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463426+0000","last_change":"2023-11-24T06:20:56.459307+0000","last_active":"2023-11-24T06:20:56.463426+0000","last_peered":"2023-11-24T06:20:56.463426+0000","last_clean":"2023-11-24T06:20:56.463426+0000","last_became_active":"2023-11-24T06:20:56.459077+0000","last_became_peered":"2023-11-24T06:20:56.459077+0000","last_unstale":"2023-11-24T06:20:56.463426+0000","last_undegraded":"2023-11-24T06:20:56.463426+0000","last_fullsized":"2023-11-24T06:20:56.463426+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:07.713106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445945+0000","last_change":"2023-11-24T06:20:56.445945+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445945+0000","last_undegraded":"2023-11-24T06:20:56.445945+0000","last_fullsized":"2023-11-24T06:20:56.445945+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461147+0000","last_change":"2023-11-24T06:20:56.457538+0000","last_active":"2023-11-24T06:20:56.461147+0000","last_peered":"2023-11-24T06:20:56.461147+0000","last_clean":"2023-11-24T06:20:56.461147+0000","last_became_active":"2023-11-24T06:20:56.457247+0000","last_became_peered":"2023-11-24T06:20:56.457247+0000","last_unstale":"2023-11-24T06:20:56.461147+0000","last_undegraded":"2023-11-24T06:20:56.461147+0000","last_fullsized":"2023-11-24T06:20:56.461147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:50.332536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445239+0000","last_change":"2023-11-24T06:20:56.445239+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445239+0000","last_undegraded":"2023-11-24T06:20:56.445239+0000","last_fullsized":"2023-11-24T06:20:56.445239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.457746+0000","last_change":"2023-11-24T06:20:56.453032+0000","last_active":"2023-11-24T06:20:56.457746+0000","last_peered":"2023-11-24T06:20:56.457746+0000","last_clean":"2023-11-24T06:20:56.457746+0000","last_became_active":"2023-11-24T06:20:56.452748+0000","last_became_peered":"2023-11-24T06:20:56.452748+0000","last_unstale":"2023-11-24T06:20:56.457746+0000","last_undegraded":"2023-11-24T06:20:56.457746+0000","last_fullsized":"2023-11-24T06:20:56.457746+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:37.569663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441010+0000","last_change":"2023-11-24T06:20:56.441010+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441010+0000","last_undegraded":"2023-11-24T06:20:56.441010+0000","last_fullsized":"2023-11-24T06:20:56.441010+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463502+0000","last_change":"2023-11-24T06:20:56.459215+0000","last_active":"2023-11-24T06:20:56.463502+0000","last_peered":"2023-11-24T06:20:56.463502+0000","last_clean":"2023-11-24T06:20:56.463502+0000","last_became_active":"2023-11-24T06:20:56.458937+0000","last_became_peered":"2023-11-24T06:20:56.458937+0000","last_unstale":"2023-11-24T06:20:56.463502+0000","last_undegraded":"2023-11-24T06:20:56.463502+0000","last_fullsized":"2023-11-24T06:20:56.463502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:03:06.709201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440845+0000","last_change":"2023-11-24T06:20:56.440845+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440845+0000","last_undegraded":"2023-11-24T06:20:56.440845+0000","last_fullsized":"2023-11-24T06:20:56.440845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458490+0000","last_change":"2023-11-24T06:20:56.455809+0000","last_active":"2023-11-24T06:20:56.458490+0000","last_peered":"2023-11-24T06:20:56.458490+0000","last_clean":"2023-11-24T06:20:56.458490+0000","last_became_active":"2023-11-24T06:20:56.455461+0000","last_became_peered":"2023-11-24T06:20:56.455461+0000","last_unstale":"2023-11-24T06:20:56.458490+0000","last_undegraded":"2023-11-24T06:20:56.458490+0000","last_fullsized":"2023-11-24T06:20:56.458490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441040+0000","last_change":"2023-11-24T06:20:56.441040+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441040+0000","last_undegraded":"2023-11-24T06:20:56.441040+0000","last_fullsized":"2023-11-24T06:20:56.441040+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454779+0000","last_change":"2023-11-24T06:20:56.450795+0000","last_active":"2023-11-24T06:20:56.454779+0000","last_peered":"2023-11-24T06:20:56.454779+0000","last_clean":"2023-11-24T06:20:56.454779+0000","last_became_active":"2023-11-24T06:20:56.450111+0000","last_became_peered":"2023-11-24T06:20:56.450111+0000","last_unstale":"2023-11-24T06:20:56.454779+0000","last_undegraded":"2023-11-24T06:20:56.454779+0000","last_fullsized":"2023-11-24T06:20:56.454779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442840+0000","last_change":"2023-11-24T06:20:56.442840+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442840+0000","last_undegraded":"2023-11-24T06:20:56.442840+0000","last_fullsized":"2023-11-24T06:20:56.442840+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461173+0000","last_change":"2023-11-24T06:20:56.458175+0000","last_active":"2023-11-24T06:20:56.461173+0000","last_peered":"2023-11-24T06:20:56.461173+0000","last_clean":"2023-11-24T06:20:56.461173+0000","last_became_active":"2023-11-24T06:20:56.457570+0000","last_became_peered":"2023-11-24T06:20:56.457570+0000","last_unstale":"2023-11-24T06:20:56.461173+0000","last_undegraded":"2023-11-24T06:20:56.461173+0000","last_fullsized":"2023-11-24T06:20:56.461173+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:05.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444484+0000","last_change":"2023-11-24T06:20:56.444484+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444484+0000","last_undegraded":"2023-11-24T06:20:56.444484+0000","last_fullsized":"2023-11-24T06:20:56.444484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458402+0000","last_change":"2023-11-24T06:20:56.454563+0000","last_active":"2023-11-24T06:20:56.458402+0000","last_peered":"2023-11-24T06:20:56.458402+0000","last_clean":"2023-11-24T06:20:56.458402+0000","last_became_active":"2023-11-24T06:20:56.454014+0000","last_became_peered":"2023-11-24T06:20:56.454014+0000","last_unstale":"2023-11-24T06:20:56.458402+0000","last_undegraded":"2023-11-24T06:20:56.458402+0000","last_fullsized":"2023-11-24T06:20:56.458402+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:22:26.548969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439927+0000","last_change":"2023-11-24T06:20:56.439927+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439927+0000","last_undegraded":"2023-11-24T06:20:56.439927+0000","last_fullsized":"2023-11-24T06:20:56.439927+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458720+0000","last_change":"2023-11-24T06:20:56.455444+0000","last_active":"2023-11-24T06:20:56.458720+0000","last_peered":"2023-11-24T06:20:56.458720+0000","last_clean":"2023-11-24T06:20:56.458720+0000","last_became_active":"2023-11-24T06:20:56.455167+0000","last_became_peered":"2023-11-24T06:20:56.455167+0000","last_unstale":"2023-11-24T06:20:56.458720+0000","last_undegraded":"2023-11-24T06:20:56.458720+0000","last_fullsized":"2023-11-24T06:20:56.458720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:59.246341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439301+0000","last_change":"2023-11-24T06:20:56.439301+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439301+0000","last_undegraded":"2023-11-24T06:20:56.439301+0000","last_fullsized":"2023-11-24T06:20:56.439301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.457669+0000","last_change":"2023-11-24T06:20:56.453405+0000","last_active":"2023-11-24T06:20:56.457669+0000","last_peered":"2023-11-24T06:20:56.457669+0000","last_clean":"2023-11-24T06:20:56.457669+0000","last_became_active":"2023-11-24T06:20:56.453077+0000","last_became_peered":"2023-11-24T06:20:56.453077+0000","last_unstale":"2023-11-24T06:20:56.457669+0000","last_undegraded":"2023-11-24T06:20:56.457669+0000","last_fullsized":"2023-11-24T06:20:56.457669+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444847+0000","last_change":"2023-11-24T06:20:56.444847+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444847+0000","last_undegraded":"2023-11-24T06:20:56.444847+0000","last_fullsized":"2023-11-24T06:20:56.444847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458271+0000","last_change":"2023-11-24T06:20:56.455600+0000","last_active":"2023-11-24T06:20:56.458271+0000","last_peered":"2023-11-24T06:20:56.458271+0000","last_clean":"2023-11-24T06:20:56.458271+0000","last_became_active":"2023-11-24T06:20:56.455299+0000","last_became_peered":"2023-11-24T06:20:56.455299+0000","last_unstale":"2023-11-24T06:20:56.458271+0000","last_undegraded":"2023-11-24T06:20:56.458271+0000","last_fullsized":"2023-11-24T06:20:56.458271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:41:00.663712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446278+0000","last_change":"2023-11-24T06:20:56.446278+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446278+0000","last_undegraded":"2023-11-24T06:20:56.446278+0000","last_fullsized":"2023-11-24T06:20:56.446278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.456003+0000","last_change":"2023-11-24T06:20:56.453053+0000","last_active":"2023-11-24T06:20:56.456003+0000","last_peered":"2023-11-24T06:20:56.456003+0000","last_clean":"2023-11-24T06:20:56.456003+0000","last_became_active":"2023-11-24T06:20:56.452778+0000","last_became_peered":"2023-11-24T06:20:56.452778+0000","last_unstale":"2023-11-24T06:20:56.456003+0000","last_undegraded":"2023-11-24T06:20:56.456003+0000","last_fullsized":"2023-11-24T06:20:56.456003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:54.169232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444952+0000","last_change":"2023-11-24T06:20:56.444952+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444952+0000","last_undegraded":"2023-11-24T06:20:56.444952+0000","last_fullsized":"2023-11-24T06:20:56.444952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452274+0000","last_change":"2023-11-24T06:20:56.445884+0000","last_active":"2023-11-24T06:20:56.452274+0000","last_peered":"2023-11-24T06:20:56.452274+0000","last_clean":"2023-11-24T06:20:56.452274+0000","last_became_active":"2023-11-24T06:20:56.445600+0000","last_became_peered":"2023-11-24T06:20:56.445600+0000","last_unstale":"2023-11-24T06:20:56.452274+0000","last_undegraded":"2023-11-24T06:20:56.452274+0000","last_fullsized":"2023-11-24T06:20:56.452274+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445794+0000","last_change":"2023-11-24T06:20:56.445794+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445794+0000","last_undegraded":"2023-11-24T06:20:56.445794+0000","last_fullsized":"2023-11-24T06:20:56.445794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461106+0000","last_change":"2023-11-24T06:20:56.456769+0000","last_active":"2023-11-24T06:20:56.461106+0000","last_peered":"2023-11-24T06:20:56.461106+0000","last_clean":"2023-11-24T06:20:56.461106+0000","last_became_active":"2023-11-24T06:20:56.456558+0000","last_became_peered":"2023-11-24T06:20:56.456558+0000","last_unstale":"2023-11-24T06:20:56.461106+0000","last_undegraded":"2023-11-24T06:20:56.461106+0000","last_fullsized":"2023-11-24T06:20:56.461106+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:47.748375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440874+0000","last_change":"2023-11-24T06:20:56.440874+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440874+0000","last_undegraded":"2023-11-24T06:20:56.440874+0000","last_fullsized":"2023-11-24T06:20:56.440874+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461112+0000","last_change":"2023-11-24T06:20:56.456842+0000","last_active":"2023-11-24T06:20:56.461112+0000","last_peered":"2023-11-24T06:20:56.461112+0000","last_clean":"2023-11-24T06:20:56.461112+0000","last_became_active":"2023-11-24T06:20:56.456506+0000","last_became_peered":"2023-11-24T06:20:56.456506+0000","last_unstale":"2023-11-24T06:20:56.461112+0000","last_undegraded":"2023-11-24T06:20:56.461112+0000","last_fullsized":"2023-11-24T06:20:56.461112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:03:05.191510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":207,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-24T06:20:55.430113+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:55.430113+0000","last_peered":"2023-11-24T06:20:55.430113+0000","last_clean":"2023-11-24T06:20:55.430113+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:55.430113+0000","last_undegraded":"2023-11-24T06:20:55.430113+0000","last_fullsized":"2023-11-24T06:20:55.430113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28340,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343500,"statfs":{"total":96636764160,"available":96607744000,"internally_reserved":0,"allocated":1335296,"data_stored":904636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59699999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.505},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.628}]}]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.308},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.018}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.625}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.439}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.5}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.60899999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.65600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":753664,"data_stored":320543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.251},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.32800000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.37},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.314}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.54000000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.70299999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.73399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28328,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343512,"statfs":{"total":96636764160,"available":96607756288,"internally_reserved":0,"allocated":1335296,"data_stored":904636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.317},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.625}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.59499999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.251}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.38200000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54100000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.624}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.26500000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.247},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32500000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.432},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.47899999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.45300000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58699999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.36899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.40300000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.5}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.505},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.68200000000000005}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.35899999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41699999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:01.947 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:02.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.099+0000 7f0d25793700 1 Processor -- start 2023-11-24T06:21:02.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.099+0000 7f0d25793700 1 -- start start 2023-11-24T06:21:02.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d25793700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d200b2b60 0x7f0d200ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d25793700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d200cf0d0 0x7f0d200d1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d25793700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2006bc00 con 0x7f0d2015e0f0 2023-11-24T06:21:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d25793700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d200cf560 con 0x7f0d200cf0d0 2023-11-24T06:21:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d25793700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d2015e4d0 con 0x7f0d200b2b60 2023-11-24T06:21:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d200cf0d0 0x7f0d200d1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1effd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0d2015e0f0 0x7f0d200b9700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42510/0 (socket says 172.21.15.89:42510) 2023-11-24T06:21:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1effd700 1 -- 172.21.15.89:0/330722300 learned_addr learned my addr 172.21.15.89:0/330722300 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1f7fe700 1 -- 172.21.15.89:0/330722300 >> v2:172.21.15.123:3300/0 conn(0x7f0d200b2b60 msgr2=0x7f0d200ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:02.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1e7fc700 1 --2- 172.21.15.89:0/330722300 >> v2:172.21.15.123:3300/0 conn(0x7f0d200b2b60 0x7f0d200ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1f7fe700 1 --2- 172.21.15.89:0/330722300 >> v2:172.21.15.123:3300/0 conn(0x7f0d200b2b60 0x7f0d200ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1dffb700 1 -- 172.21.15.89:0/330722300 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316439393 0 0) 0x7f0d2006bc00 con 0x7f0d2015e0f0 2023-11-24T06:21:02.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1f7fe700 1 -- 172.21.15.89:0/330722300 >> v1:172.21.15.89:6789/0 conn(0x7f0d2015e0f0 legacy=0x7f0d200b9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1f7fe700 1 -- 172.21.15.89:0/330722300 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2006aff0 con 0x7f0d200cf0d0 2023-11-24T06:21:02.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1e7fc700 1 --2- 172.21.15.89:0/330722300 >> v2:172.21.15.123:3300/0 conn(0x7f0d200b2b60 0x7f0d200ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:02.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1f7fe700 1 --2- 172.21.15.89:0/330722300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d200d1f60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0d200d3cc0 tx=0x7f0d1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1f90a854288441 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.100+0000 7f0d1dffb700 1 -- 172.21.15.89:0/330722300 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1800e070 con 0x7f0d200cf0d0 2023-11-24T06:21:02.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d1dffb700 1 -- 172.21.15.89:0/330722300 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1800bbd0 con 0x7f0d200cf0d0 2023-11-24T06:21:02.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d1dffb700 1 -- 172.21.15.89:0/330722300 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18012430 con 0x7f0d200cf0d0 2023-11-24T06:21:02.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d25793700 1 -- 172.21.15.89:0/330722300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 msgr2=0x7f0d200d1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d25793700 1 --2- 172.21.15.89:0/330722300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d200d1f60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0d200d3cc0 tx=0x7f0d1800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:02.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d25793700 1 -- 172.21.15.89:0/330722300 shutdown_connections 2023-11-24T06:21:02.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d25793700 1 --2- 172.21.15.89:0/330722300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d200d1f60 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d25793700 1 --2- 172.21.15.89:0/330722300 >> v2:172.21.15.123:3300/0 conn(0x7f0d200b2b60 0x7f0d200ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d25793700 1 -- 172.21.15.89:0/330722300 >> 172.21.15.89:0/330722300 conn(0x7f0d200b5ce0 msgr2=0x7f0d200b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:02.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d25793700 1 -- 172.21.15.89:0/330722300 shutdown_connections 2023-11-24T06:21:02.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.101+0000 7f0d25793700 1 -- 172.21.15.89:0/330722300 wait complete. 2023-11-24T06:21:02.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.102+0000 7f0d25793700 1 Processor -- start 2023-11-24T06:21:02.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.102+0000 7f0d25793700 1 -- start start 2023-11-24T06:21:02.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d25793700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d200b2b60 0x7f0d2014e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d25793700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d20161080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d25793700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2015e0f0 0x7f0d201f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d25793700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200c97d0 con 0x7f0d200cf0d0 2023-11-24T06:21:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d25793700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d201586f0 con 0x7f0d200b2b60 2023-11-24T06:21:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d25793700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200c8b50 con 0x7f0d2015e0f0 2023-11-24T06:21:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d1e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d20161080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d1e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d20161080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39014/0 (socket says 172.21.15.89:39014) 2023-11-24T06:21:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d1e7fc700 1 -- 172.21.15.89:0/1153997151 learned_addr learned my addr 172.21.15.89:0/1153997151 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d1effd700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d200b2b60 0x7f0d2014e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.103+0000 7f0d1f7fe700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2015e0f0 0x7f0d201f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d1e7fc700 1 -- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2015e0f0 msgr2=0x7f0d201f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d1e7fc700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2015e0f0 0x7f0d201f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d1e7fc700 1 -- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d200b2b60 msgr2=0x7f0d2014e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d1e7fc700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d200b2b60 0x7f0d2014e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d1e7fc700 1 -- 172.21.15.89:0/1153997151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d18009a70 con 0x7f0d200cf0d0 2023-11-24T06:21:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d1effd700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d200b2b60 0x7f0d2014e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d1e7fc700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d20161080 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c750 tx=0x7f0d1000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d077fe700 1 -- 172.21.15.89:0/1153997151 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10011800 con 0x7f0d200cf0d0 2023-11-24T06:21:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d077fe700 1 -- 172.21.15.89:0/1153997151 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d10011960 con 0x7f0d200cf0d0 2023-11-24T06:21:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d200c8e30 con 0x7f0d200cf0d0 2023-11-24T06:21:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d077fe700 1 -- 172.21.15.89:0/1153997151 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10011cd0 con 0x7f0d200cf0d0 2023-11-24T06:21:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.104+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d200c93f0 con 0x7f0d200cf0d0 2023-11-24T06:21:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.105+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d200c9910 con 0x7f0d200cf0d0 2023-11-24T06:21:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.105+0000 7f0d077fe700 1 -- 172.21.15.89:0/1153997151 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f0d10021350 con 0x7f0d200cf0d0 2023-11-24T06:21:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.106+0000 7f0d077fe700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d0c06af60 0x7f0d0c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.106+0000 7f0d077fe700 1 -- 172.21.15.89:0/1153997151 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d10089720 con 0x7f0d200cf0d0 2023-11-24T06:21:02.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.106+0000 7f0d1effd700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d0c06af60 0x7f0d0c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.106+0000 7f0d1effd700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d0c06af60 0x7f0d0c06d420 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0d14000f60 tx=0x7f0d14007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.110+0000 7f0d077fe700 1 -- 172.21.15.89:0/1153997151 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d200c9910 con 0x7f0d200cf0d0 2023-11-24T06:21:02.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.297+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d2006b890 con 0x7f0d0c06af60 2023-11-24T06:21:02.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.304+0000 7f0d077fe700 1 -- 172.21.15.89:0/1153997151 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284020 (secure 0 0 0) 0x7f0d14004870 con 0x7f0d0c06af60 2023-11-24T06:21:02.309 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:02.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d0c06af60 msgr2=0x7f0d0c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d0c06af60 0x7f0d0c06d420 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0d14000f60 tx=0x7f0d14007d30 comp rx=0 tx=0).stop 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 msgr2=0x7f0d20161080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d20161080 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c750 tx=0x7f0d1000cb10 comp rx=0 tx=0).stop 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 shutdown_connections 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2015e0f0 0x7f0d201f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d0c06af60 0x7f0d0c06d420 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d200b2b60 0x7f0d2014e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 --2- 172.21.15.89:0/1153997151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d200cf0d0 0x7f0d20161080 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.307+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 >> 172.21.15.89:0/1153997151 conn(0x7f0d200b5ce0 msgr2=0x7f0d20161b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.308+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 shutdown_connections 2023-11-24T06:21:02.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.308+0000 7f0d25793700 1 -- 172.21.15.89:0/1153997151 wait complete. 2023-11-24T06:21:02.313 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:02.321 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2023-11-24T06:21:01.745460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223148,"kb_used_data":6964,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751572,"statfs":{"total":773094113280,"available":772865609728,"internally_reserved":0,"allocated":7131136,"data_stored":3701155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001006"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441539+0000","last_change":"2023-11-24T06:20:56.441539+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441539+0000","last_undegraded":"2023-11-24T06:20:56.441539+0000","last_fullsized":"2023-11-24T06:20:56.441539+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449390+0000","last_change":"2023-11-24T06:20:56.449390+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449390+0000","last_undegraded":"2023-11-24T06:20:56.449390+0000","last_fullsized":"2023-11-24T06:20:56.449390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441773+0000","last_change":"2023-11-24T06:20:56.441773+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441773+0000","last_undegraded":"2023-11-24T06:20:56.441773+0000","last_fullsized":"2023-11-24T06:20:56.441773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446842+0000","last_change":"2023-11-24T06:20:56.446842+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446842+0000","last_undegraded":"2023-11-24T06:20:56.446842+0000","last_fullsized":"2023-11-24T06:20:56.446842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445464+0000","last_change":"2023-11-24T06:20:56.445464+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445464+0000","last_undegraded":"2023-11-24T06:20:56.445464+0000","last_fullsized":"2023-11-24T06:20:56.445464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.450162+0000","last_change":"2023-11-24T06:20:56.450162+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.450162+0000","last_undegraded":"2023-11-24T06:20:56.450162+0000","last_fullsized":"2023-11-24T06:20:56.450162+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446689+0000","last_change":"2023-11-24T06:20:56.446689+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446689+0000","last_undegraded":"2023-11-24T06:20:56.446689+0000","last_fullsized":"2023-11-24T06:20:56.446689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449944+0000","last_change":"2023-11-24T06:20:56.449944+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449944+0000","last_undegraded":"2023-11-24T06:20:56.449944+0000","last_fullsized":"2023-11-24T06:20:56.449944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.447505+0000","last_change":"2023-11-24T06:20:56.447505+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.447505+0000","last_undegraded":"2023-11-24T06:20:56.447505+0000","last_fullsized":"2023-11-24T06:20:56.447505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443357+0000","last_change":"2023-11-24T06:20:56.443357+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443357+0000","last_undegraded":"2023-11-24T06:20:56.443357+0000","last_fullsized":"2023-11-24T06:20:56.443357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440281+0000","last_change":"2023-11-24T06:20:56.440281+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440281+0000","last_undegraded":"2023-11-24T06:20:56.440281+0000","last_fullsized":"2023-11-24T06:20:56.440281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446432+0000","last_change":"2023-11-24T06:20:56.446432+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446432+0000","last_undegraded":"2023-11-24T06:20:56.446432+0000","last_fullsized":"2023-11-24T06:20:56.446432+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442484+0000","last_change":"2023-11-24T06:20:56.442484+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442484+0000","last_undegraded":"2023-11-24T06:20:56.442484+0000","last_fullsized":"2023-11-24T06:20:56.442484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445693+0000","last_change":"2023-11-24T06:20:56.445693+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445693+0000","last_undegraded":"2023-11-24T06:20:56.445693+0000","last_fullsized":"2023-11-24T06:20:56.445693+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443169+0000","last_change":"2023-11-24T06:20:56.443169+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443169+0000","last_undegraded":"2023-11-24T06:20:56.443169+0000","last_fullsized":"2023-11-24T06:20:56.443169+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443905+0000","last_change":"2023-11-24T06:20:56.443905+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443905+0000","last_undegraded":"2023-11-24T06:20:56.443905+0000","last_fullsized":"2023-11-24T06:20:56.443905+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446697+0000","last_change":"2023-11-24T06:20:56.446697+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446697+0000","last_undegraded":"2023-11-24T06:20:56.446697+0000","last_fullsized":"2023-11-24T06:20:56.446697+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444979+0000","last_change":"2023-11-24T06:20:56.444979+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444979+0000","last_undegraded":"2023-11-24T06:20:56.444979+0000","last_fullsized":"2023-11-24T06:20:56.444979+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.451453+0000","last_change":"2023-11-24T06:20:56.451453+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.451453+0000","last_undegraded":"2023-11-24T06:20:56.451453+0000","last_fullsized":"2023-11-24T06:20:56.451453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442861+0000","last_change":"2023-11-24T06:20:56.442861+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442861+0000","last_undegraded":"2023-11-24T06:20:56.442861+0000","last_fullsized":"2023-11-24T06:20:56.442861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443493+0000","last_change":"2023-11-24T06:20:56.443493+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443493+0000","last_undegraded":"2023-11-24T06:20:56.443493+0000","last_fullsized":"2023-11-24T06:20:56.443493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443677+0000","last_change":"2023-11-24T06:20:56.443677+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443677+0000","last_undegraded":"2023-11-24T06:20:56.443677+0000","last_fullsized":"2023-11-24T06:20:56.443677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438156+0000","last_change":"2023-11-24T06:20:56.438156+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438156+0000","last_undegraded":"2023-11-24T06:20:56.438156+0000","last_fullsized":"2023-11-24T06:20:56.438156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438903+0000","last_change":"2023-11-24T06:20:56.438903+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438903+0000","last_undegraded":"2023-11-24T06:20:56.438903+0000","last_fullsized":"2023-11-24T06:20:56.438903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443185+0000","last_change":"2023-11-24T06:20:56.443185+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443185+0000","last_undegraded":"2023-11-24T06:20:56.443185+0000","last_fullsized":"2023-11-24T06:20:56.443185+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446196+0000","last_change":"2023-11-24T06:20:56.446196+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446196+0000","last_undegraded":"2023-11-24T06:20:56.446196+0000","last_fullsized":"2023-11-24T06:20:56.446196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441252+0000","last_change":"2023-11-24T06:20:56.441252+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441252+0000","last_undegraded":"2023-11-24T06:20:56.441252+0000","last_fullsized":"2023-11-24T06:20:56.441252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438658+0000","last_change":"2023-11-24T06:20:56.438658+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438658+0000","last_undegraded":"2023-11-24T06:20:56.438658+0000","last_fullsized":"2023-11-24T06:20:56.438658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441558+0000","last_change":"2023-11-24T06:20:56.441558+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441558+0000","last_undegraded":"2023-11-24T06:20:56.441558+0000","last_fullsized":"2023-11-24T06:20:56.441558+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445126+0000","last_change":"2023-11-24T06:20:56.445126+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445126+0000","last_undegraded":"2023-11-24T06:20:56.445126+0000","last_fullsized":"2023-11-24T06:20:56.445126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446281+0000","last_change":"2023-11-24T06:20:56.446281+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446281+0000","last_undegraded":"2023-11-24T06:20:56.446281+0000","last_fullsized":"2023-11-24T06:20:56.446281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449025+0000","last_change":"2023-11-24T06:20:56.449025+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449025+0000","last_undegraded":"2023-11-24T06:20:56.449025+0000","last_fullsized":"2023-11-24T06:20:56.449025+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440692+0000","last_change":"2023-11-24T06:20:56.440692+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440692+0000","last_undegraded":"2023-11-24T06:20:56.440692+0000","last_fullsized":"2023-11-24T06:20:56.440692+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.455924+0000","last_change":"2023-11-24T06:20:56.450056+0000","last_active":"2023-11-24T06:20:56.455924+0000","last_peered":"2023-11-24T06:20:56.455924+0000","last_clean":"2023-11-24T06:20:56.455924+0000","last_became_active":"2023-11-24T06:20:56.449760+0000","last_became_peered":"2023-11-24T06:20:56.449760+0000","last_unstale":"2023-11-24T06:20:56.455924+0000","last_undegraded":"2023-11-24T06:20:56.455924+0000","last_fullsized":"2023-11-24T06:20:56.455924+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:42.412869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445988+0000","last_change":"2023-11-24T06:20:56.445988+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445988+0000","last_undegraded":"2023-11-24T06:20:56.445988+0000","last_fullsized":"2023-11-24T06:20:56.445988+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454721+0000","last_change":"2023-11-24T06:20:56.450515+0000","last_active":"2023-11-24T06:20:56.454721+0000","last_peered":"2023-11-24T06:20:56.454721+0000","last_clean":"2023-11-24T06:20:56.454721+0000","last_became_active":"2023-11-24T06:20:56.450219+0000","last_became_peered":"2023-11-24T06:20:56.450219+0000","last_unstale":"2023-11-24T06:20:56.454721+0000","last_undegraded":"2023-11-24T06:20:56.454721+0000","last_fullsized":"2023-11-24T06:20:56.454721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:40:15.263523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449285+0000","last_change":"2023-11-24T06:20:56.449285+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449285+0000","last_undegraded":"2023-11-24T06:20:56.449285+0000","last_fullsized":"2023-11-24T06:20:56.449285+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463396+0000","last_change":"2023-11-24T06:20:56.457862+0000","last_active":"2023-11-24T06:20:56.463396+0000","last_peered":"2023-11-24T06:20:56.463396+0000","last_clean":"2023-11-24T06:20:56.463396+0000","last_became_active":"2023-11-24T06:20:56.457485+0000","last_became_peered":"2023-11-24T06:20:56.457485+0000","last_unstale":"2023-11-24T06:20:56.463396+0000","last_undegraded":"2023-11-24T06:20:56.463396+0000","last_fullsized":"2023-11-24T06:20:56.463396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:44:24.187407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439756+0000","last_change":"2023-11-24T06:20:56.439756+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439756+0000","last_undegraded":"2023-11-24T06:20:56.439756+0000","last_fullsized":"2023-11-24T06:20:56.439756+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461058+0000","last_change":"2023-11-24T06:20:56.456592+0000","last_active":"2023-11-24T06:20:56.461058+0000","last_peered":"2023-11-24T06:20:56.461058+0000","last_clean":"2023-11-24T06:20:56.461058+0000","last_became_active":"2023-11-24T06:20:56.456416+0000","last_became_peered":"2023-11-24T06:20:56.456416+0000","last_unstale":"2023-11-24T06:20:56.461058+0000","last_undegraded":"2023-11-24T06:20:56.461058+0000","last_fullsized":"2023-11-24T06:20:56.461058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:19:08.950567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444793+0000","last_change":"2023-11-24T06:20:56.444793+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444793+0000","last_undegraded":"2023-11-24T06:20:56.444793+0000","last_fullsized":"2023-11-24T06:20:56.444793+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.455514+0000","last_change":"2023-11-24T06:20:56.452696+0000","last_active":"2023-11-24T06:20:56.455514+0000","last_peered":"2023-11-24T06:20:56.455514+0000","last_clean":"2023-11-24T06:20:56.455514+0000","last_became_active":"2023-11-24T06:20:56.452226+0000","last_became_peered":"2023-11-24T06:20:56.452226+0000","last_unstale":"2023-11-24T06:20:56.455514+0000","last_undegraded":"2023-11-24T06:20:56.455514+0000","last_fullsized":"2023-11-24T06:20:56.455514+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:59:01.228879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438821+0000","last_change":"2023-11-24T06:20:56.438821+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438821+0000","last_undegraded":"2023-11-24T06:20:56.438821+0000","last_fullsized":"2023-11-24T06:20:56.438821+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463517+0000","last_change":"2023-11-24T06:20:56.459294+0000","last_active":"2023-11-24T06:20:56.463517+0000","last_peered":"2023-11-24T06:20:56.463517+0000","last_clean":"2023-11-24T06:20:56.463517+0000","last_became_active":"2023-11-24T06:20:56.458979+0000","last_became_peered":"2023-11-24T06:20:56.458979+0000","last_unstale":"2023-11-24T06:20:56.463517+0000","last_undegraded":"2023-11-24T06:20:56.463517+0000","last_fullsized":"2023-11-24T06:20:56.463517+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:26.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446512+0000","last_change":"2023-11-24T06:20:56.446512+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446512+0000","last_undegraded":"2023-11-24T06:20:56.446512+0000","last_fullsized":"2023-11-24T06:20:56.446512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454765+0000","last_change":"2023-11-24T06:20:56.451454+0000","last_active":"2023-11-24T06:20:56.454765+0000","last_peered":"2023-11-24T06:20:56.454765+0000","last_clean":"2023-11-24T06:20:56.454765+0000","last_became_active":"2023-11-24T06:20:56.451064+0000","last_became_peered":"2023-11-24T06:20:56.451064+0000","last_unstale":"2023-11-24T06:20:56.454765+0000","last_undegraded":"2023-11-24T06:20:56.454765+0000","last_fullsized":"2023-11-24T06:20:56.454765+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442438+0000","last_change":"2023-11-24T06:20:56.442438+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442438+0000","last_undegraded":"2023-11-24T06:20:56.442438+0000","last_fullsized":"2023-11-24T06:20:56.442438+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454770+0000","last_change":"2023-11-24T06:20:56.449997+0000","last_active":"2023-11-24T06:20:56.454770+0000","last_peered":"2023-11-24T06:20:56.454770+0000","last_clean":"2023-11-24T06:20:56.454770+0000","last_became_active":"2023-11-24T06:20:56.449727+0000","last_became_peered":"2023-11-24T06:20:56.449727+0000","last_unstale":"2023-11-24T06:20:56.454770+0000","last_undegraded":"2023-11-24T06:20:56.454770+0000","last_fullsized":"2023-11-24T06:20:56.454770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:13.838572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445798+0000","last_change":"2023-11-24T06:20:56.445798+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445798+0000","last_undegraded":"2023-11-24T06:20:56.445798+0000","last_fullsized":"2023-11-24T06:20:56.445798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452073+0000","last_change":"2023-11-24T06:20:56.444835+0000","last_active":"2023-11-24T06:20:56.452073+0000","last_peered":"2023-11-24T06:20:56.452073+0000","last_clean":"2023-11-24T06:20:56.452073+0000","last_became_active":"2023-11-24T06:20:56.444547+0000","last_became_peered":"2023-11-24T06:20:56.444547+0000","last_unstale":"2023-11-24T06:20:56.452073+0000","last_undegraded":"2023-11-24T06:20:56.452073+0000","last_fullsized":"2023-11-24T06:20:56.452073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:12.584029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441093+0000","last_change":"2023-11-24T06:20:56.441093+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441093+0000","last_undegraded":"2023-11-24T06:20:56.441093+0000","last_fullsized":"2023-11-24T06:20:56.441093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458584+0000","last_change":"2023-11-24T06:20:56.455466+0000","last_active":"2023-11-24T06:20:56.458584+0000","last_peered":"2023-11-24T06:20:56.458584+0000","last_clean":"2023-11-24T06:20:56.458584+0000","last_became_active":"2023-11-24T06:20:56.455137+0000","last_became_peered":"2023-11-24T06:20:56.455137+0000","last_unstale":"2023-11-24T06:20:56.458584+0000","last_undegraded":"2023-11-24T06:20:56.458584+0000","last_fullsized":"2023-11-24T06:20:56.458584+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:25:31.853530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446530+0000","last_change":"2023-11-24T06:20:56.446530+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446530+0000","last_undegraded":"2023-11-24T06:20:56.446530+0000","last_fullsized":"2023-11-24T06:20:56.446530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.456617+0000","last_change":"2023-11-24T06:20:56.454452+0000","last_active":"2023-11-24T06:20:56.456617+0000","last_peered":"2023-11-24T06:20:56.456617+0000","last_clean":"2023-11-24T06:20:56.456617+0000","last_became_active":"2023-11-24T06:20:56.454143+0000","last_became_peered":"2023-11-24T06:20:56.454143+0000","last_unstale":"2023-11-24T06:20:56.456617+0000","last_undegraded":"2023-11-24T06:20:56.456617+0000","last_fullsized":"2023-11-24T06:20:56.456617+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:31.618289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445684+0000","last_change":"2023-11-24T06:20:56.445684+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445684+0000","last_undegraded":"2023-11-24T06:20:56.445684+0000","last_fullsized":"2023-11-24T06:20:56.445684+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454599+0000","last_change":"2023-11-24T06:20:56.450022+0000","last_active":"2023-11-24T06:20:56.454599+0000","last_peered":"2023-11-24T06:20:56.454599+0000","last_clean":"2023-11-24T06:20:56.454599+0000","last_became_active":"2023-11-24T06:20:56.449589+0000","last_became_peered":"2023-11-24T06:20:56.449589+0000","last_unstale":"2023-11-24T06:20:56.454599+0000","last_undegraded":"2023-11-24T06:20:56.454599+0000","last_fullsized":"2023-11-24T06:20:56.454599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438775+0000","last_change":"2023-11-24T06:20:56.438775+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438775+0000","last_undegraded":"2023-11-24T06:20:56.438775+0000","last_fullsized":"2023-11-24T06:20:56.438775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452082+0000","last_change":"2023-11-24T06:20:56.448978+0000","last_active":"2023-11-24T06:20:56.452082+0000","last_peered":"2023-11-24T06:20:56.452082+0000","last_clean":"2023-11-24T06:20:56.452082+0000","last_became_active":"2023-11-24T06:20:56.448657+0000","last_became_peered":"2023-11-24T06:20:56.448657+0000","last_unstale":"2023-11-24T06:20:56.452082+0000","last_undegraded":"2023-11-24T06:20:56.452082+0000","last_fullsized":"2023-11-24T06:20:56.452082+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:12.584029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443962+0000","last_change":"2023-11-24T06:20:56.443962+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443962+0000","last_undegraded":"2023-11-24T06:20:56.443962+0000","last_fullsized":"2023-11-24T06:20:56.443962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458629+0000","last_change":"2023-11-24T06:20:56.455610+0000","last_active":"2023-11-24T06:20:56.458629+0000","last_peered":"2023-11-24T06:20:56.458629+0000","last_clean":"2023-11-24T06:20:56.458629+0000","last_became_active":"2023-11-24T06:20:56.455289+0000","last_became_peered":"2023-11-24T06:20:56.455289+0000","last_unstale":"2023-11-24T06:20:56.458629+0000","last_undegraded":"2023-11-24T06:20:56.458629+0000","last_fullsized":"2023-11-24T06:20:56.458629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:17.452542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442464+0000","last_change":"2023-11-24T06:20:56.442464+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442464+0000","last_undegraded":"2023-11-24T06:20:56.442464+0000","last_fullsized":"2023-11-24T06:20:56.442464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458511+0000","last_change":"2023-11-24T06:20:56.455905+0000","last_active":"2023-11-24T06:20:56.458511+0000","last_peered":"2023-11-24T06:20:56.458511+0000","last_clean":"2023-11-24T06:20:56.458511+0000","last_became_active":"2023-11-24T06:20:56.455373+0000","last_became_peered":"2023-11-24T06:20:56.455373+0000","last_unstale":"2023-11-24T06:20:56.458511+0000","last_undegraded":"2023-11-24T06:20:56.458511+0000","last_fullsized":"2023-11-24T06:20:56.458511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446537+0000","last_change":"2023-11-24T06:20:56.446537+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446537+0000","last_undegraded":"2023-11-24T06:20:56.446537+0000","last_fullsized":"2023-11-24T06:20:56.446537+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452266+0000","last_change":"2023-11-24T06:20:56.445784+0000","last_active":"2023-11-24T06:20:56.452266+0000","last_peered":"2023-11-24T06:20:56.452266+0000","last_clean":"2023-11-24T06:20:56.452266+0000","last_became_active":"2023-11-24T06:20:56.445468+0000","last_became_peered":"2023-11-24T06:20:56.445468+0000","last_unstale":"2023-11-24T06:20:56.452266+0000","last_undegraded":"2023-11-24T06:20:56.452266+0000","last_fullsized":"2023-11-24T06:20:56.452266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442896+0000","last_change":"2023-11-24T06:20:56.442896+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442896+0000","last_undegraded":"2023-11-24T06:20:56.442896+0000","last_fullsized":"2023-11-24T06:20:56.442896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461188+0000","last_change":"2023-11-24T06:20:56.458379+0000","last_active":"2023-11-24T06:20:56.461188+0000","last_peered":"2023-11-24T06:20:56.461188+0000","last_clean":"2023-11-24T06:20:56.461188+0000","last_became_active":"2023-11-24T06:20:56.457956+0000","last_became_peered":"2023-11-24T06:20:56.457956+0000","last_unstale":"2023-11-24T06:20:56.461188+0000","last_undegraded":"2023-11-24T06:20:56.461188+0000","last_fullsized":"2023-11-24T06:20:56.461188+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:20.310219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440565+0000","last_change":"2023-11-24T06:20:56.440565+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440565+0000","last_undegraded":"2023-11-24T06:20:56.440565+0000","last_fullsized":"2023-11-24T06:20:56.440565+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463426+0000","last_change":"2023-11-24T06:20:56.459307+0000","last_active":"2023-11-24T06:20:56.463426+0000","last_peered":"2023-11-24T06:20:56.463426+0000","last_clean":"2023-11-24T06:20:56.463426+0000","last_became_active":"2023-11-24T06:20:56.459077+0000","last_became_peered":"2023-11-24T06:20:56.459077+0000","last_unstale":"2023-11-24T06:20:56.463426+0000","last_undegraded":"2023-11-24T06:20:56.463426+0000","last_fullsized":"2023-11-24T06:20:56.463426+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:07.713106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445945+0000","last_change":"2023-11-24T06:20:56.445945+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445945+0000","last_undegraded":"2023-11-24T06:20:56.445945+0000","last_fullsized":"2023-11-24T06:20:56.445945+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461147+0000","last_change":"2023-11-24T06:20:56.457538+0000","last_active":"2023-11-24T06:20:56.461147+0000","last_peered":"2023-11-24T06:20:56.461147+0000","last_clean":"2023-11-24T06:20:56.461147+0000","last_became_active":"2023-11-24T06:20:56.457247+0000","last_became_peered":"2023-11-24T06:20:56.457247+0000","last_unstale":"2023-11-24T06:20:56.461147+0000","last_undegraded":"2023-11-24T06:20:56.461147+0000","last_fullsized":"2023-11-24T06:20:56.461147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:50.332536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445239+0000","last_change":"2023-11-24T06:20:56.445239+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445239+0000","last_undegraded":"2023-11-24T06:20:56.445239+0000","last_fullsized":"2023-11-24T06:20:56.445239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.457746+0000","last_change":"2023-11-24T06:20:56.453032+0000","last_active":"2023-11-24T06:20:56.457746+0000","last_peered":"2023-11-24T06:20:56.457746+0000","last_clean":"2023-11-24T06:20:56.457746+0000","last_became_active":"2023-11-24T06:20:56.452748+0000","last_became_peered":"2023-11-24T06:20:56.452748+0000","last_unstale":"2023-11-24T06:20:56.457746+0000","last_undegraded":"2023-11-24T06:20:56.457746+0000","last_fullsized":"2023-11-24T06:20:56.457746+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:37.569663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441010+0000","last_change":"2023-11-24T06:20:56.441010+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441010+0000","last_undegraded":"2023-11-24T06:20:56.441010+0000","last_fullsized":"2023-11-24T06:20:56.441010+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463502+0000","last_change":"2023-11-24T06:20:56.459215+0000","last_active":"2023-11-24T06:20:56.463502+0000","last_peered":"2023-11-24T06:20:56.463502+0000","last_clean":"2023-11-24T06:20:56.463502+0000","last_became_active":"2023-11-24T06:20:56.458937+0000","last_became_peered":"2023-11-24T06:20:56.458937+0000","last_unstale":"2023-11-24T06:20:56.463502+0000","last_undegraded":"2023-11-24T06:20:56.463502+0000","last_fullsized":"2023-11-24T06:20:56.463502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:03:06.709201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440845+0000","last_change":"2023-11-24T06:20:56.440845+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440845+0000","last_undegraded":"2023-11-24T06:20:56.440845+0000","last_fullsized":"2023-11-24T06:20:56.440845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458490+0000","last_change":"2023-11-24T06:20:56.455809+0000","last_active":"2023-11-24T06:20:56.458490+0000","last_peered":"2023-11-24T06:20:56.458490+0000","last_clean":"2023-11-24T06:20:56.458490+0000","last_became_active":"2023-11-24T06:20:56.455461+0000","last_became_peered":"2023-11-24T06:20:56.455461+0000","last_unstale":"2023-11-24T06:20:56.458490+0000","last_undegraded":"2023-11-24T06:20:56.458490+0000","last_fullsized":"2023-11-24T06:20:56.458490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441040+0000","last_change":"2023-11-24T06:20:56.441040+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441040+0000","last_undegraded":"2023-11-24T06:20:56.441040+0000","last_fullsized":"2023-11-24T06:20:56.441040+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454779+0000","last_change":"2023-11-24T06:20:56.450795+0000","last_active":"2023-11-24T06:20:56.454779+0000","last_peered":"2023-11-24T06:20:56.454779+0000","last_clean":"2023-11-24T06:20:56.454779+0000","last_became_active":"2023-11-24T06:20:56.450111+0000","last_became_peered":"2023-11-24T06:20:56.450111+0000","last_unstale":"2023-11-24T06:20:56.454779+0000","last_undegraded":"2023-11-24T06:20:56.454779+0000","last_fullsized":"2023-11-24T06:20:56.454779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442840+0000","last_change":"2023-11-24T06:20:56.442840+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442840+0000","last_undegraded":"2023-11-24T06:20:56.442840+0000","last_fullsized":"2023-11-24T06:20:56.442840+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461173+0000","last_change":"2023-11-24T06:20:56.458175+0000","last_active":"2023-11-24T06:20:56.461173+0000","last_peered":"2023-11-24T06:20:56.461173+0000","last_clean":"2023-11-24T06:20:56.461173+0000","last_became_active":"2023-11-24T06:20:56.457570+0000","last_became_peered":"2023-11-24T06:20:56.457570+0000","last_unstale":"2023-11-24T06:20:56.461173+0000","last_undegraded":"2023-11-24T06:20:56.461173+0000","last_fullsized":"2023-11-24T06:20:56.461173+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:05.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444484+0000","last_change":"2023-11-24T06:20:56.444484+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444484+0000","last_undegraded":"2023-11-24T06:20:56.444484+0000","last_fullsized":"2023-11-24T06:20:56.444484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458402+0000","last_change":"2023-11-24T06:20:56.454563+0000","last_active":"2023-11-24T06:20:56.458402+0000","last_peered":"2023-11-24T06:20:56.458402+0000","last_clean":"2023-11-24T06:20:56.458402+0000","last_became_active":"2023-11-24T06:20:56.454014+0000","last_became_peered":"2023-11-24T06:20:56.454014+0000","last_unstale":"2023-11-24T06:20:56.458402+0000","last_undegraded":"2023-11-24T06:20:56.458402+0000","last_fullsized":"2023-11-24T06:20:56.458402+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:22:26.548969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439927+0000","last_change":"2023-11-24T06:20:56.439927+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439927+0000","last_undegraded":"2023-11-24T06:20:56.439927+0000","last_fullsized":"2023-11-24T06:20:56.439927+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458720+0000","last_change":"2023-11-24T06:20:56.455444+0000","last_active":"2023-11-24T06:20:56.458720+0000","last_peered":"2023-11-24T06:20:56.458720+0000","last_clean":"2023-11-24T06:20:56.458720+0000","last_became_active":"2023-11-24T06:20:56.455167+0000","last_became_peered":"2023-11-24T06:20:56.455167+0000","last_unstale":"2023-11-24T06:20:56.458720+0000","last_undegraded":"2023-11-24T06:20:56.458720+0000","last_fullsized":"2023-11-24T06:20:56.458720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:59.246341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439301+0000","last_change":"2023-11-24T06:20:56.439301+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439301+0000","last_undegraded":"2023-11-24T06:20:56.439301+0000","last_fullsized":"2023-11-24T06:20:56.439301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.457669+0000","last_change":"2023-11-24T06:20:56.453405+0000","last_active":"2023-11-24T06:20:56.457669+0000","last_peered":"2023-11-24T06:20:56.457669+0000","last_clean":"2023-11-24T06:20:56.457669+0000","last_became_active":"2023-11-24T06:20:56.453077+0000","last_became_peered":"2023-11-24T06:20:56.453077+0000","last_unstale":"2023-11-24T06:20:56.457669+0000","last_undegraded":"2023-11-24T06:20:56.457669+0000","last_fullsized":"2023-11-24T06:20:56.457669+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444847+0000","last_change":"2023-11-24T06:20:56.444847+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444847+0000","last_undegraded":"2023-11-24T06:20:56.444847+0000","last_fullsized":"2023-11-24T06:20:56.444847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458271+0000","last_change":"2023-11-24T06:20:56.455600+0000","last_active":"2023-11-24T06:20:56.458271+0000","last_peered":"2023-11-24T06:20:56.458271+0000","last_clean":"2023-11-24T06:20:56.458271+0000","last_became_active":"2023-11-24T06:20:56.455299+0000","last_became_peered":"2023-11-24T06:20:56.455299+0000","last_unstale":"2023-11-24T06:20:56.458271+0000","last_undegraded":"2023-11-24T06:20:56.458271+0000","last_fullsized":"2023-11-24T06:20:56.458271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:41:00.663712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446278+0000","last_change":"2023-11-24T06:20:56.446278+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446278+0000","last_undegraded":"2023-11-24T06:20:56.446278+0000","last_fullsized":"2023-11-24T06:20:56.446278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.456003+0000","last_change":"2023-11-24T06:20:56.453053+0000","last_active":"2023-11-24T06:20:56.456003+0000","last_peered":"2023-11-24T06:20:56.456003+0000","last_clean":"2023-11-24T06:20:56.456003+0000","last_became_active":"2023-11-24T06:20:56.452778+0000","last_became_peered":"2023-11-24T06:20:56.452778+0000","last_unstale":"2023-11-24T06:20:56.456003+0000","last_undegraded":"2023-11-24T06:20:56.456003+0000","last_fullsized":"2023-11-24T06:20:56.456003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:54.169232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444952+0000","last_change":"2023-11-24T06:20:56.444952+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444952+0000","last_undegraded":"2023-11-24T06:20:56.444952+0000","last_fullsized":"2023-11-24T06:20:56.444952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452274+0000","last_change":"2023-11-24T06:20:56.445884+0000","last_active":"2023-11-24T06:20:56.452274+0000","last_peered":"2023-11-24T06:20:56.452274+0000","last_clean":"2023-11-24T06:20:56.452274+0000","last_became_active":"2023-11-24T06:20:56.445600+0000","last_became_peered":"2023-11-24T06:20:56.445600+0000","last_unstale":"2023-11-24T06:20:56.452274+0000","last_undegraded":"2023-11-24T06:20:56.452274+0000","last_fullsized":"2023-11-24T06:20:56.452274+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445794+0000","last_change":"2023-11-24T06:20:56.445794+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445794+0000","last_undegraded":"2023-11-24T06:20:56.445794+0000","last_fullsized":"2023-11-24T06:20:56.445794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461106+0000","last_change":"2023-11-24T06:20:56.456769+0000","last_active":"2023-11-24T06:20:56.461106+0000","last_peered":"2023-11-24T06:20:56.461106+0000","last_clean":"2023-11-24T06:20:56.461106+0000","last_became_active":"2023-11-24T06:20:56.456558+0000","last_became_peered":"2023-11-24T06:20:56.456558+0000","last_unstale":"2023-11-24T06:20:56.461106+0000","last_undegraded":"2023-11-24T06:20:56.461106+0000","last_fullsized":"2023-11-24T06:20:56.461106+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:47.748375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440874+0000","last_change":"2023-11-24T06:20:56.440874+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440874+0000","last_undegraded":"2023-11-24T06:20:56.440874+0000","last_fullsized":"2023-11-24T06:20:56.440874+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461112+0000","last_change":"2023-11-24T06:20:56.456842+0000","last_active":"2023-11-24T06:20:56.461112+0000","last_peered":"2023-11-24T06:20:56.461112+0000","last_clean":"2023-11-24T06:20:56.461112+0000","last_became_active":"2023-11-24T06:20:56.456506+0000","last_became_peered":"2023-11-24T06:20:56.456506+0000","last_unstale":"2023-11-24T06:20:56.461112+0000","last_undegraded":"2023-11-24T06:20:56.461112+0000","last_fullsized":"2023-11-24T06:20:56.461112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:03:05.191510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":207,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-24T06:20:55.430113+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:55.430113+0000","last_peered":"2023-11-24T06:20:55.430113+0000","last_clean":"2023-11-24T06:20:55.430113+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:55.430113+0000","last_undegraded":"2023-11-24T06:20:55.430113+0000","last_fullsized":"2023-11-24T06:20:55.430113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28340,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343500,"statfs":{"total":96636764160,"available":96607744000,"internally_reserved":0,"allocated":1335296,"data_stored":904636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59699999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.505},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.628}]}]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.308},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.018}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.625}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.439}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.5}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.60899999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.65600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":753664,"data_stored":320543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.251},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.32800000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.37},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.314}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.54000000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.70299999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.73399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28328,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343512,"statfs":{"total":96636764160,"available":96607756288,"internally_reserved":0,"allocated":1335296,"data_stored":904636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.317},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.625}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.59499999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.251}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.38200000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54100000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.624}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.26500000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.247},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32500000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.432},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.47899999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.45300000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58699999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.36899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.40300000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.5}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.505},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.68200000000000005}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.35899999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41699999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:02.326 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3f is not active+clean 2023-11-24T06:21:02.326 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.441539+0000', 'last_change': '2023-11-24T06:20:56.441539+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.441539+0000', 'last_undegraded': '2023-11-24T06:20:56.441539+0000', 'last_fullsized': '2023-11-24T06:20:56.441539+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.326 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3e is not active+clean 2023-11-24T06:21:02.326 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.449390+0000', 'last_change': '2023-11-24T06:20:56.449390+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.449390+0000', 'last_undegraded': '2023-11-24T06:20:56.449390+0000', 'last_fullsized': '2023-11-24T06:20:56.449390+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3d is not active+clean 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.441773+0000', 'last_change': '2023-11-24T06:20:56.441773+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.441773+0000', 'last_undegraded': '2023-11-24T06:20:56.441773+0000', 'last_fullsized': '2023-11-24T06:20:56.441773+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3c is not active+clean 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446842+0000', 'last_change': '2023-11-24T06:20:56.446842+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446842+0000', 'last_undegraded': '2023-11-24T06:20:56.446842+0000', 'last_fullsized': '2023-11-24T06:20:56.446842+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3b is not active+clean 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445464+0000', 'last_change': '2023-11-24T06:20:56.445464+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445464+0000', 'last_undegraded': '2023-11-24T06:20:56.445464+0000', 'last_fullsized': '2023-11-24T06:20:56.445464+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3a is not active+clean 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.450162+0000', 'last_change': '2023-11-24T06:20:56.450162+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.450162+0000', 'last_undegraded': '2023-11-24T06:20:56.450162+0000', 'last_fullsized': '2023-11-24T06:20:56.450162+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.39 is not active+clean 2023-11-24T06:21:02.327 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446689+0000', 'last_change': '2023-11-24T06:20:56.446689+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446689+0000', 'last_undegraded': '2023-11-24T06:20:56.446689+0000', 'last_fullsized': '2023-11-24T06:20:56.446689+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.38 is not active+clean 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.449944+0000', 'last_change': '2023-11-24T06:20:56.449944+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.449944+0000', 'last_undegraded': '2023-11-24T06:20:56.449944+0000', 'last_fullsized': '2023-11-24T06:20:56.449944+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.37 is not active+clean 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.447505+0000', 'last_change': '2023-11-24T06:20:56.447505+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.447505+0000', 'last_undegraded': '2023-11-24T06:20:56.447505+0000', 'last_fullsized': '2023-11-24T06:20:56.447505+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.36 is not active+clean 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.443357+0000', 'last_change': '2023-11-24T06:20:56.443357+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.443357+0000', 'last_undegraded': '2023-11-24T06:20:56.443357+0000', 'last_fullsized': '2023-11-24T06:20:56.443357+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.35 is not active+clean 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.440281+0000', 'last_change': '2023-11-24T06:20:56.440281+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.440281+0000', 'last_undegraded': '2023-11-24T06:20:56.440281+0000', 'last_fullsized': '2023-11-24T06:20:56.440281+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.34 is not active+clean 2023-11-24T06:21:02.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446432+0000', 'last_change': '2023-11-24T06:20:56.446432+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446432+0000', 'last_undegraded': '2023-11-24T06:20:56.446432+0000', 'last_fullsized': '2023-11-24T06:20:56.446432+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.33 is not active+clean 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.442484+0000', 'last_change': '2023-11-24T06:20:56.442484+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.442484+0000', 'last_undegraded': '2023-11-24T06:20:56.442484+0000', 'last_fullsized': '2023-11-24T06:20:56.442484+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.32 is not active+clean 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445693+0000', 'last_change': '2023-11-24T06:20:56.445693+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445693+0000', 'last_undegraded': '2023-11-24T06:20:56.445693+0000', 'last_fullsized': '2023-11-24T06:20:56.445693+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.31 is not active+clean 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.443169+0000', 'last_change': '2023-11-24T06:20:56.443169+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.443169+0000', 'last_undegraded': '2023-11-24T06:20:56.443169+0000', 'last_fullsized': '2023-11-24T06:20:56.443169+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.30 is not active+clean 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.443905+0000', 'last_change': '2023-11-24T06:20:56.443905+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.443905+0000', 'last_undegraded': '2023-11-24T06:20:56.443905+0000', 'last_fullsized': '2023-11-24T06:20:56.443905+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2f is not active+clean 2023-11-24T06:21:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446697+0000', 'last_change': '2023-11-24T06:20:56.446697+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446697+0000', 'last_undegraded': '2023-11-24T06:20:56.446697+0000', 'last_fullsized': '2023-11-24T06:20:56.446697+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2e is not active+clean 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.444979+0000', 'last_change': '2023-11-24T06:20:56.444979+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.444979+0000', 'last_undegraded': '2023-11-24T06:20:56.444979+0000', 'last_fullsized': '2023-11-24T06:20:56.444979+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2d is not active+clean 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.451453+0000', 'last_change': '2023-11-24T06:20:56.451453+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.451453+0000', 'last_undegraded': '2023-11-24T06:20:56.451453+0000', 'last_fullsized': '2023-11-24T06:20:56.451453+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2c is not active+clean 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.442861+0000', 'last_change': '2023-11-24T06:20:56.442861+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.442861+0000', 'last_undegraded': '2023-11-24T06:20:56.442861+0000', 'last_fullsized': '2023-11-24T06:20:56.442861+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2b is not active+clean 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.443493+0000', 'last_change': '2023-11-24T06:20:56.443493+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.443493+0000', 'last_undegraded': '2023-11-24T06:20:56.443493+0000', 'last_fullsized': '2023-11-24T06:20:56.443493+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:21:02.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2a is not active+clean 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.443677+0000', 'last_change': '2023-11-24T06:20:56.443677+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.443677+0000', 'last_undegraded': '2023-11-24T06:20:56.443677+0000', 'last_fullsized': '2023-11-24T06:20:56.443677+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.29 is not active+clean 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.438156+0000', 'last_change': '2023-11-24T06:20:56.438156+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.438156+0000', 'last_undegraded': '2023-11-24T06:20:56.438156+0000', 'last_fullsized': '2023-11-24T06:20:56.438156+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.28 is not active+clean 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.438903+0000', 'last_change': '2023-11-24T06:20:56.438903+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.438903+0000', 'last_undegraded': '2023-11-24T06:20:56.438903+0000', 'last_fullsized': '2023-11-24T06:20:56.438903+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.27 is not active+clean 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.443185+0000', 'last_change': '2023-11-24T06:20:56.443185+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.443185+0000', 'last_undegraded': '2023-11-24T06:20:56.443185+0000', 'last_fullsized': '2023-11-24T06:20:56.443185+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.26 is not active+clean 2023-11-24T06:21:02.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446196+0000', 'last_change': '2023-11-24T06:20:56.446196+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446196+0000', 'last_undegraded': '2023-11-24T06:20:56.446196+0000', 'last_fullsized': '2023-11-24T06:20:56.446196+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.25 is not active+clean 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.441252+0000', 'last_change': '2023-11-24T06:20:56.441252+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.441252+0000', 'last_undegraded': '2023-11-24T06:20:56.441252+0000', 'last_fullsized': '2023-11-24T06:20:56.441252+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.24 is not active+clean 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.438658+0000', 'last_change': '2023-11-24T06:20:56.438658+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.438658+0000', 'last_undegraded': '2023-11-24T06:20:56.438658+0000', 'last_fullsized': '2023-11-24T06:20:56.438658+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.23 is not active+clean 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.441558+0000', 'last_change': '2023-11-24T06:20:56.441558+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.441558+0000', 'last_undegraded': '2023-11-24T06:20:56.441558+0000', 'last_fullsized': '2023-11-24T06:20:56.441558+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.22 is not active+clean 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445126+0000', 'last_change': '2023-11-24T06:20:56.445126+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445126+0000', 'last_undegraded': '2023-11-24T06:20:56.445126+0000', 'last_fullsized': '2023-11-24T06:20:56.445126+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:21:02.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.21 is not active+clean 2023-11-24T06:21:02.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446281+0000', 'last_change': '2023-11-24T06:20:56.446281+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446281+0000', 'last_undegraded': '2023-11-24T06:20:56.446281+0000', 'last_fullsized': '2023-11-24T06:20:56.446281+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:21:02.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.20 is not active+clean 2023-11-24T06:21:02.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.449025+0000', 'last_change': '2023-11-24T06:20:56.449025+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.449025+0000', 'last_undegraded': '2023-11-24T06:20:56.449025+0000', 'last_fullsized': '2023-11-24T06:20:56.449025+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2023-11-24T06:21:02.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.440692+0000', 'last_change': '2023-11-24T06:20:56.440692+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.440692+0000', 'last_undegraded': '2023-11-24T06:20:56.440692+0000', 'last_fullsized': '2023-11-24T06:20:56.440692+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2023-11-24T06:21:02.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445988+0000', 'last_change': '2023-11-24T06:20:56.445988+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445988+0000', 'last_undegraded': '2023-11-24T06:20:56.445988+0000', 'last_fullsized': '2023-11-24T06:20:56.445988+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.449285+0000', 'last_change': '2023-11-24T06:20:56.449285+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.449285+0000', 'last_undegraded': '2023-11-24T06:20:56.449285+0000', 'last_fullsized': '2023-11-24T06:20:56.449285+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.439756+0000', 'last_change': '2023-11-24T06:20:56.439756+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.439756+0000', 'last_undegraded': '2023-11-24T06:20:56.439756+0000', 'last_fullsized': '2023-11-24T06:20:56.439756+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.444793+0000', 'last_change': '2023-11-24T06:20:56.444793+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.444793+0000', 'last_undegraded': '2023-11-24T06:20:56.444793+0000', 'last_fullsized': '2023-11-24T06:20:56.444793+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.438821+0000', 'last_change': '2023-11-24T06:20:56.438821+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.438821+0000', 'last_undegraded': '2023-11-24T06:20:56.438821+0000', 'last_fullsized': '2023-11-24T06:20:56.438821+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2023-11-24T06:21:02.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446512+0000', 'last_change': '2023-11-24T06:20:56.446512+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446512+0000', 'last_undegraded': '2023-11-24T06:20:56.446512+0000', 'last_fullsized': '2023-11-24T06:20:56.446512+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.442438+0000', 'last_change': '2023-11-24T06:20:56.442438+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.442438+0000', 'last_undegraded': '2023-11-24T06:20:56.442438+0000', 'last_fullsized': '2023-11-24T06:20:56.442438+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445798+0000', 'last_change': '2023-11-24T06:20:56.445798+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445798+0000', 'last_undegraded': '2023-11-24T06:20:56.445798+0000', 'last_fullsized': '2023-11-24T06:20:56.445798+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.441093+0000', 'last_change': '2023-11-24T06:20:56.441093+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.441093+0000', 'last_undegraded': '2023-11-24T06:20:56.441093+0000', 'last_fullsized': '2023-11-24T06:20:56.441093+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446530+0000', 'last_change': '2023-11-24T06:20:56.446530+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446530+0000', 'last_undegraded': '2023-11-24T06:20:56.446530+0000', 'last_fullsized': '2023-11-24T06:20:56.446530+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2023-11-24T06:21:02.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445684+0000', 'last_change': '2023-11-24T06:20:56.445684+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445684+0000', 'last_undegraded': '2023-11-24T06:20:56.445684+0000', 'last_fullsized': '2023-11-24T06:20:56.445684+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.438775+0000', 'last_change': '2023-11-24T06:20:56.438775+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.438775+0000', 'last_undegraded': '2023-11-24T06:20:56.438775+0000', 'last_fullsized': '2023-11-24T06:20:56.438775+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.443962+0000', 'last_change': '2023-11-24T06:20:56.443962+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.443962+0000', 'last_undegraded': '2023-11-24T06:20:56.443962+0000', 'last_fullsized': '2023-11-24T06:20:56.443962+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.442464+0000', 'last_change': '2023-11-24T06:20:56.442464+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.442464+0000', 'last_undegraded': '2023-11-24T06:20:56.442464+0000', 'last_fullsized': '2023-11-24T06:20:56.442464+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446537+0000', 'last_change': '2023-11-24T06:20:56.446537+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446537+0000', 'last_undegraded': '2023-11-24T06:20:56.446537+0000', 'last_fullsized': '2023-11-24T06:20:56.446537+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2023-11-24T06:21:02.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.442896+0000', 'last_change': '2023-11-24T06:20:56.442896+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.442896+0000', 'last_undegraded': '2023-11-24T06:20:56.442896+0000', 'last_fullsized': '2023-11-24T06:20:56.442896+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.440565+0000', 'last_change': '2023-11-24T06:20:56.440565+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.440565+0000', 'last_undegraded': '2023-11-24T06:20:56.440565+0000', 'last_fullsized': '2023-11-24T06:20:56.440565+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445945+0000', 'last_change': '2023-11-24T06:20:56.445945+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445945+0000', 'last_undegraded': '2023-11-24T06:20:56.445945+0000', 'last_fullsized': '2023-11-24T06:20:56.445945+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445239+0000', 'last_change': '2023-11-24T06:20:56.445239+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445239+0000', 'last_undegraded': '2023-11-24T06:20:56.445239+0000', 'last_fullsized': '2023-11-24T06:20:56.445239+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.441010+0000', 'last_change': '2023-11-24T06:20:56.441010+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.441010+0000', 'last_undegraded': '2023-11-24T06:20:56.441010+0000', 'last_fullsized': '2023-11-24T06:20:56.441010+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:21:02.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2023-11-24T06:21:02.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.440845+0000', 'last_change': '2023-11-24T06:20:56.440845+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.440845+0000', 'last_undegraded': '2023-11-24T06:20:56.440845+0000', 'last_fullsized': '2023-11-24T06:20:56.440845+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:21:02.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2023-11-24T06:21:02.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.441040+0000', 'last_change': '2023-11-24T06:20:56.441040+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.441040+0000', 'last_undegraded': '2023-11-24T06:20:56.441040+0000', 'last_fullsized': '2023-11-24T06:20:56.441040+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:21:02.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2023-11-24T06:21:02.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.442840+0000', 'last_change': '2023-11-24T06:20:56.442840+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.442840+0000', 'last_undegraded': '2023-11-24T06:20:56.442840+0000', 'last_fullsized': '2023-11-24T06:20:56.442840+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-24T06:21:02.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2023-11-24T06:21:02.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.444484+0000', 'last_change': '2023-11-24T06:20:56.444484+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.444484+0000', 'last_undegraded': '2023-11-24T06:20:56.444484+0000', 'last_fullsized': '2023-11-24T06:20:56.444484+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.439927+0000', 'last_change': '2023-11-24T06:20:56.439927+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.439927+0000', 'last_undegraded': '2023-11-24T06:20:56.439927+0000', 'last_fullsized': '2023-11-24T06:20:56.439927+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.439301+0000', 'last_change': '2023-11-24T06:20:56.439301+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.439301+0000', 'last_undegraded': '2023-11-24T06:20:56.439301+0000', 'last_fullsized': '2023-11-24T06:20:56.439301+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.444847+0000', 'last_change': '2023-11-24T06:20:56.444847+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.444847+0000', 'last_undegraded': '2023-11-24T06:20:56.444847+0000', 'last_fullsized': '2023-11-24T06:20:56.444847+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.446278+0000', 'last_change': '2023-11-24T06:20:56.446278+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.446278+0000', 'last_undegraded': '2023-11-24T06:20:56.446278+0000', 'last_fullsized': '2023-11-24T06:20:56.446278+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2023-11-24T06:21:02.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.444952+0000', 'last_change': '2023-11-24T06:20:56.444952+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.444952+0000', 'last_undegraded': '2023-11-24T06:20:56.444952+0000', 'last_fullsized': '2023-11-24T06:20:56.444952+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T06:21:02.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2023-11-24T06:21:02.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.445794+0000', 'last_change': '2023-11-24T06:20:56.445794+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.445794+0000', 'last_undegraded': '2023-11-24T06:20:56.445794+0000', 'last_fullsized': '2023-11-24T06:20:56.445794+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-11-24T06:21:02.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2023-11-24T06:21:02.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 62, 'state': 'creating+peering', 'last_fresh': '2023-11-24T06:20:56.440874+0000', 'last_change': '2023-11-24T06:20:56.440874+0000', 'last_active': '2023-11-24T06:20:56.425380+0000', 'last_peered': '2023-11-24T06:20:56.425380+0000', 'last_clean': '2023-11-24T06:20:56.425380+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-24T06:20:56.440874+0000', 'last_undegraded': '2023-11-24T06:20:56.440874+0000', 'last_fullsized': '2023-11-24T06:20:56.440874+0000', 'mapping_epoch': 62, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 62, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'last_clean_scrub_stamp': '2023-11-24T06:20:56.425380+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-24T06:21:02.340 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-11-24T06:21:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.480+0000 7f9b30873700 1 Processor -- start 2023-11-24T06:21:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.481+0000 7f9b30873700 1 -- start start 2023-11-24T06:21:02.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.481+0000 7f9b30873700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9b2c1446a0 0x7f9b2c0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.481+0000 7f9b30873700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c0af950 con 0x7f9b2c0adf00 2023-11-24T06:21:02.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.481+0000 7f9b30873700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c0afc30 con 0x7f9b2c06c710 2023-11-24T06:21:02.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.481+0000 7f9b30873700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2c144a80 con 0x7f9b2c1446a0 2023-11-24T06:21:02.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b29d9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9b2c0adf00 0x7f9b2c06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42526/0 (socket says 172.21.15.89:42526) 2023-11-24T06:21:02.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b29d9b700 1 -- 172.21.15.89:0/389386297 learned_addr learned my addr 172.21.15.89:0/389386297 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:02.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2a59c700 1 --2- 172.21.15.89:0/389386297 >> v2:172.21.15.123:3300/0 conn(0x7f9b2c1446a0 0x7f9b2c0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2959a700 1 -- 172.21.15.89:0/389386297 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198748405 0 0) 0x7f9b2c0af950 con 0x7f9b2c0adf00 2023-11-24T06:21:02.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2959a700 1 -- 172.21.15.89:0/389386297 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b180035b0 con 0x7f9b2c0adf00 2023-11-24T06:21:02.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2a59c700 1 -- 172.21.15.89:0/389386297 >> v1:172.21.15.123:6789/0 conn(0x7f9b2c06c710 legacy=0x7f9b2c0b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2a59c700 1 -- 172.21.15.89:0/389386297 >> v1:172.21.15.89:6789/0 conn(0x7f9b2c0adf00 legacy=0x7f9b2c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2a59c700 1 -- 172.21.15.89:0/389386297 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0aff10 con 0x7f9b2c1446a0 2023-11-24T06:21:02.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2a59c700 1 --2- 172.21.15.89:0/389386297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 0x7f9b2c0b5ae0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009d90 tx=0x7f9b1c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ab7e80714faf11 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2959a700 1 -- 172.21.15.89:0/389386297 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c00e070 con 0x7f9b2c1446a0 2023-11-24T06:21:02.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2959a700 1 -- 172.21.15.89:0/389386297 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c00bb20 con 0x7f9b2c1446a0 2023-11-24T06:21:02.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.482+0000 7f9b2959a700 1 -- 172.21.15.89:0/389386297 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c012430 con 0x7f9b2c1446a0 2023-11-24T06:21:02.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.483+0000 7f9b30873700 1 -- 172.21.15.89:0/389386297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 msgr2=0x7f9b2c0b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.483+0000 7f9b30873700 1 --2- 172.21.15.89:0/389386297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 0x7f9b2c0b5ae0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009d90 tx=0x7f9b1c00b330 comp rx=0 tx=0).stop 2023-11-24T06:21:02.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.483+0000 7f9b30873700 1 -- 172.21.15.89:0/389386297 shutdown_connections 2023-11-24T06:21:02.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.483+0000 7f9b30873700 1 --2- 172.21.15.89:0/389386297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 0x7f9b2c0b5ae0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.483+0000 7f9b30873700 1 -- 172.21.15.89:0/389386297 >> 172.21.15.89:0/389386297 conn(0x7f9b2c071b90 msgr2=0x7f9b2c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:02.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.483+0000 7f9b30873700 1 -- 172.21.15.89:0/389386297 shutdown_connections 2023-11-24T06:21:02.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.483+0000 7f9b30873700 1 -- 172.21.15.89:0/389386297 wait complete. 2023-11-24T06:21:02.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.484+0000 7f9b30873700 1 Processor -- start 2023-11-24T06:21:02.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.484+0000 7f9b30873700 1 -- start start 2023-11-24T06:21:02.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b30873700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b2c06c710 0x7f9b2c13a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b30873700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b2c0adf00 0x7f9b2c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b30873700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 0x7f9b2c0cee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b30873700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c15cd50 con 0x7f9b2c06c710 2023-11-24T06:21:02.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b30873700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c15bcb0 con 0x7f9b2c1446a0 2023-11-24T06:21:02.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b30873700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c15bf90 con 0x7f9b2c0adf00 2023-11-24T06:21:02.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b2a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b2c06c710 0x7f9b2c13a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b2a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b2c06c710 0x7f9b2c13a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39026/0 (socket says 172.21.15.89:39026) 2023-11-24T06:21:02.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b2a59c700 1 -- 172.21.15.89:0/3966322156 learned_addr learned my addr 172.21.15.89:0/3966322156 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:02.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b29d9b700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b2c0adf00 0x7f9b2c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.485+0000 7f9b2ad9d700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 0x7f9b2c0cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b2a59c700 1 -- 172.21.15.89:0/3966322156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b2c0adf00 msgr2=0x7f9b2c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b2a59c700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b2c0adf00 0x7f9b2c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b2a59c700 1 -- 172.21.15.89:0/3966322156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 msgr2=0x7f9b2c0cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b2a59c700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 0x7f9b2c0cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b2a59c700 1 -- 172.21.15.89:0/3966322156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c009a70 con 0x7f9b2c06c710 2023-11-24T06:21:02.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b2a59c700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b2c06c710 0x7f9b2c13a3b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c15b310 tx=0x7f9b1c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b22ffd700 1 -- 172.21.15.89:0/3966322156 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c012430 con 0x7f9b2c06c710 2023-11-24T06:21:02.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b22ffd700 1 -- 172.21.15.89:0/3966322156 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c020070 con 0x7f9b2c06c710 2023-11-24T06:21:02.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2c15c270 con 0x7f9b2c06c710 2023-11-24T06:21:02.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b22ffd700 1 -- 172.21.15.89:0/3966322156 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c004280 con 0x7f9b2c06c710 2023-11-24T06:21:02.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.486+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2c15c830 con 0x7f9b2c06c710 2023-11-24T06:21:02.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.487+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2c0753f0 con 0x7f9b2c06c710 2023-11-24T06:21:02.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.487+0000 7f9b22ffd700 1 -- 172.21.15.89:0/3966322156 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f9b1c0043e0 con 0x7f9b2c06c710 2023-11-24T06:21:02.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.488+0000 7f9b22ffd700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b0c06af60 0x7f9b0c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.488+0000 7f9b22ffd700 1 -- 172.21.15.89:0/3966322156 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b1c022070 con 0x7f9b2c06c710 2023-11-24T06:21:02.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.488+0000 7f9b29d9b700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b0c06af60 0x7f9b0c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.488+0000 7f9b29d9b700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b0c06af60 0x7f9b0c06d420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c15d370 tx=0x7f9b14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.492+0000 7f9b22ffd700 1 -- 172.21.15.89:0/3966322156 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b1c0564c0 con 0x7f9b2c06c710 2023-11-24T06:21:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.763+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f9b2c067f60 con 0x7f9b2c06c710 2023-11-24T06:21:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.763+0000 7f9b22ffd700 1 -- 172.21.15.89:0/3966322156 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1452 (secure 0 0 0) 0x7f9b1c059ae0 con 0x7f9b2c06c710 2023-11-24T06:21:02.766 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:02.766 INFO:teuthology.orchestra.run.smithi089.stdout:{"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":158,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":64,"num_osds":8,"num_up_osds":8,"osd_up_since":1700806715,"num_in_osds":8,"osd_in_since":1700806706,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":228503552,"bytes_avail":772865609728,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":41,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"b","status":"up:active","gid":6039}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2023-11-24T06:20:45.743035+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.765+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b0c06af60 msgr2=0x7f9b0c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.765+0000 7f9b30873700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b0c06af60 0x7f9b0c06d420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c15d370 tx=0x7f9b14009040 comp rx=0 tx=0).stop 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b2c06c710 msgr2=0x7f9b2c13a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b2c06c710 0x7f9b2c13a3b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c15b310 tx=0x7f9b1c004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 shutdown_connections 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9b0c06af60 0x7f9b0c06d420 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b2c1446a0 0x7f9b2c0cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b2c06c710 0x7f9b2c13a3b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 --2- 172.21.15.89:0/3966322156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b2c0adf00 0x7f9b2c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 >> 172.21.15.89:0/3966322156 conn(0x7f9b2c071b90 msgr2=0x7f9b2c06b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 shutdown_connections 2023-11-24T06:21:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.766+0000 7f9b30873700 1 -- 172.21.15.89:0/3966322156 wait complete. 2023-11-24T06:21:02.780 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-11-24T06:21:02.780 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:02.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.898+0000 7fb045c67700 1 Processor -- start 2023-11-24T06:21:02.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.898+0000 7fb045c67700 1 -- start start 2023-11-24T06:21:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb045c67700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb040152cc0 0x7fb04006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb045c67700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 0x7fb04006c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb045c67700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04015c4b0 con 0x7fb0400cff20 2023-11-24T06:21:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb045c67700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04015c790 con 0x7fb040152cc0 2023-11-24T06:21:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb045c67700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04015ca70 con 0x7fb0400b49a0 2023-11-24T06:21:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb040152cc0 0x7fb04006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb040152cc0 0x7fb04006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39034/0 (socket says 172.21.15.89:39034) 2023-11-24T06:21:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03f7fe700 1 -- 172.21.15.89:0/359581107 learned_addr learned my addr 172.21.15.89:0/359581107 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03effd700 1 --2- 172.21.15.89:0/359581107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 0x7fb04006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03f7fe700 1 -- 172.21.15.89:0/359581107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 msgr2=0x7fb04006c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03f7fe700 1 --2- 172.21.15.89:0/359581107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 0x7fb04006c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03f7fe700 1 -- 172.21.15.89:0/359581107 >> v1:172.21.15.123:6789/0 conn(0x7fb0400cff20 legacy=0x7fb0400d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03f7fe700 1 -- 172.21.15.89:0/359581107 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04015cd50 con 0x7fb040152cc0 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03f7fe700 1 --2- 172.21.15.89:0/359581107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb040152cc0 0x7fb04006c000 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098f0 tx=0x7fb030009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94111d3d8eeb4fb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03e7fc700 1 -- 172.21.15.89:0/359581107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03000b8a0 con 0x7fb040152cc0 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03e7fc700 1 -- 172.21.15.89:0/359581107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03000ba00 con 0x7fb040152cc0 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.899+0000 7fb03e7fc700 1 -- 172.21.15.89:0/359581107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03000bd90 con 0x7fb040152cc0 2023-11-24T06:21:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.900+0000 7fb045c67700 1 -- 172.21.15.89:0/359581107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb040152cc0 msgr2=0x7fb04006c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.900+0000 7fb045c67700 1 --2- 172.21.15.89:0/359581107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb040152cc0 0x7fb04006c000 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098f0 tx=0x7fb030009c00 comp rx=0 tx=0).stop 2023-11-24T06:21:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.900+0000 7fb045c67700 1 -- 172.21.15.89:0/359581107 shutdown_connections 2023-11-24T06:21:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.900+0000 7fb045c67700 1 --2- 172.21.15.89:0/359581107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb040152cc0 0x7fb04006c000 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.900+0000 7fb045c67700 1 --2- 172.21.15.89:0/359581107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 0x7fb04006c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.900+0000 7fb045c67700 1 -- 172.21.15.89:0/359581107 >> 172.21.15.89:0/359581107 conn(0x7fb040071b90 msgr2=0x7fb040071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.900+0000 7fb045c67700 1 -- 172.21.15.89:0/359581107 shutdown_connections 2023-11-24T06:21:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.900+0000 7fb045c67700 1 -- 172.21.15.89:0/359581107 wait complete. 2023-11-24T06:21:02.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.901+0000 7fb045c67700 1 Processor -- start 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.901+0000 7fb045c67700 1 -- start start 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb045c67700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 0x7fb040147710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb045c67700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0400cff20 0x7fb0401532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb045c67700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb040152cc0 0x7fb04006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb045c67700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040155470 con 0x7fb0400cff20 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb045c67700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040155750 con 0x7fb040152cc0 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb045c67700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb040155a30 con 0x7fb0400b49a0 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0400cff20 0x7fb0401532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0400cff20 0x7fb0401532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39042/0 (socket says 172.21.15.89:39042) 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03effd700 1 -- 172.21.15.89:0/2025433037 learned_addr learned my addr 172.21.15.89:0/2025433037 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03f7fe700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 0x7fb040147710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03ffff700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb040152cc0 0x7fb04006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03effd700 1 -- 172.21.15.89:0/2025433037 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 msgr2=0x7fb040147710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03effd700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 0x7fb040147710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03effd700 1 -- 172.21.15.89:0/2025433037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb040152cc0 msgr2=0x7fb04006b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03effd700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb040152cc0 0x7fb04006b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.902+0000 7fb03effd700 1 -- 172.21.15.89:0/2025433037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030009550 con 0x7fb0400cff20 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.903+0000 7fb03effd700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0400cff20 0x7fb0401532b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb03400e810 tx=0x7fb03400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.903+0000 7fb027fff700 1 -- 172.21.15.89:0/2025433037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034013610 con 0x7fb0400cff20 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.903+0000 7fb027fff700 1 -- 172.21.15.89:0/2025433037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034013770 con 0x7fb0400cff20 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.903+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb040155d10 con 0x7fb0400cff20 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.903+0000 7fb027fff700 1 -- 172.21.15.89:0/2025433037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034013b20 con 0x7fb0400cff20 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.903+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04015f250 con 0x7fb0400cff20 2023-11-24T06:21:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.904+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0400753f0 con 0x7fb0400cff20 2023-11-24T06:21:02.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.904+0000 7fb027fff700 1 -- 172.21.15.89:0/2025433037 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fb034013c80 con 0x7fb0400cff20 2023-11-24T06:21:02.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.904+0000 7fb027fff700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb02c06af60 0x7fb02c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:02.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.905+0000 7fb027fff700 1 -- 172.21.15.89:0/2025433037 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb034089460 con 0x7fb0400cff20 2023-11-24T06:21:02.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.905+0000 7fb03f7fe700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb02c06af60 0x7fb02c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:02.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.905+0000 7fb03f7fe700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb02c06af60 0x7fb02c06d420 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb030014040 tx=0x7fb030008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:02.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:02.909+0000 7fb027fff700 1 -- 172.21.15.89:0/2025433037 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb034054420 con 0x7fb0400cff20 2023-11-24T06:21:03.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.094+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0401558c0 con 0x7fb02c06af60 2023-11-24T06:21:03.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.102+0000 7fb027fff700 1 -- 172.21.15.89:0/2025433037 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284020 (secure 0 0 0) 0x7fb0401558c0 con 0x7fb02c06af60 2023-11-24T06:21:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:03.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb02c06af60 msgr2=0x7fb02c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb02c06af60 0x7fb02c06d420 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb030014040 tx=0x7fb030008030 comp rx=0 tx=0).stop 2023-11-24T06:21:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0400cff20 msgr2=0x7fb0401532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0400cff20 0x7fb0401532b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb03400e810 tx=0x7fb03400ebd0 comp rx=0 tx=0).stop 2023-11-24T06:21:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 shutdown_connections 2023-11-24T06:21:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb02c06af60 0x7fb02c06d420 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:03.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb040152cc0 0x7fb04006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0400b49a0 0x7fb040147710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 --2- 172.21.15.89:0/2025433037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0400cff20 0x7fb0401532b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 >> 172.21.15.89:0/2025433037 conn(0x7fb040071b90 msgr2=0x7fb0400b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 shutdown_connections 2023-11-24T06:21:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:03.105+0000 7fb045c67700 1 -- 172.21.15.89:0/2025433037 wait complete. 2023-11-24T06:21:03.111 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:03.119 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2023-11-24T06:21:01.745460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223148,"kb_used_data":6964,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751572,"statfs":{"total":773094113280,"available":772865609728,"internally_reserved":0,"allocated":7131136,"data_stored":3701155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001006"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441539+0000","last_change":"2023-11-24T06:20:56.441539+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441539+0000","last_undegraded":"2023-11-24T06:20:56.441539+0000","last_fullsized":"2023-11-24T06:20:56.441539+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449390+0000","last_change":"2023-11-24T06:20:56.449390+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449390+0000","last_undegraded":"2023-11-24T06:20:56.449390+0000","last_fullsized":"2023-11-24T06:20:56.449390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441773+0000","last_change":"2023-11-24T06:20:56.441773+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441773+0000","last_undegraded":"2023-11-24T06:20:56.441773+0000","last_fullsized":"2023-11-24T06:20:56.441773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446842+0000","last_change":"2023-11-24T06:20:56.446842+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446842+0000","last_undegraded":"2023-11-24T06:20:56.446842+0000","last_fullsized":"2023-11-24T06:20:56.446842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445464+0000","last_change":"2023-11-24T06:20:56.445464+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445464+0000","last_undegraded":"2023-11-24T06:20:56.445464+0000","last_fullsized":"2023-11-24T06:20:56.445464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.450162+0000","last_change":"2023-11-24T06:20:56.450162+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.450162+0000","last_undegraded":"2023-11-24T06:20:56.450162+0000","last_fullsized":"2023-11-24T06:20:56.450162+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446689+0000","last_change":"2023-11-24T06:20:56.446689+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446689+0000","last_undegraded":"2023-11-24T06:20:56.446689+0000","last_fullsized":"2023-11-24T06:20:56.446689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449944+0000","last_change":"2023-11-24T06:20:56.449944+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449944+0000","last_undegraded":"2023-11-24T06:20:56.449944+0000","last_fullsized":"2023-11-24T06:20:56.449944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.447505+0000","last_change":"2023-11-24T06:20:56.447505+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.447505+0000","last_undegraded":"2023-11-24T06:20:56.447505+0000","last_fullsized":"2023-11-24T06:20:56.447505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443357+0000","last_change":"2023-11-24T06:20:56.443357+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443357+0000","last_undegraded":"2023-11-24T06:20:56.443357+0000","last_fullsized":"2023-11-24T06:20:56.443357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440281+0000","last_change":"2023-11-24T06:20:56.440281+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440281+0000","last_undegraded":"2023-11-24T06:20:56.440281+0000","last_fullsized":"2023-11-24T06:20:56.440281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446432+0000","last_change":"2023-11-24T06:20:56.446432+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446432+0000","last_undegraded":"2023-11-24T06:20:56.446432+0000","last_fullsized":"2023-11-24T06:20:56.446432+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442484+0000","last_change":"2023-11-24T06:20:56.442484+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442484+0000","last_undegraded":"2023-11-24T06:20:56.442484+0000","last_fullsized":"2023-11-24T06:20:56.442484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445693+0000","last_change":"2023-11-24T06:20:56.445693+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445693+0000","last_undegraded":"2023-11-24T06:20:56.445693+0000","last_fullsized":"2023-11-24T06:20:56.445693+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443169+0000","last_change":"2023-11-24T06:20:56.443169+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443169+0000","last_undegraded":"2023-11-24T06:20:56.443169+0000","last_fullsized":"2023-11-24T06:20:56.443169+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443905+0000","last_change":"2023-11-24T06:20:56.443905+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443905+0000","last_undegraded":"2023-11-24T06:20:56.443905+0000","last_fullsized":"2023-11-24T06:20:56.443905+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446697+0000","last_change":"2023-11-24T06:20:56.446697+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446697+0000","last_undegraded":"2023-11-24T06:20:56.446697+0000","last_fullsized":"2023-11-24T06:20:56.446697+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444979+0000","last_change":"2023-11-24T06:20:56.444979+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444979+0000","last_undegraded":"2023-11-24T06:20:56.444979+0000","last_fullsized":"2023-11-24T06:20:56.444979+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.451453+0000","last_change":"2023-11-24T06:20:56.451453+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.451453+0000","last_undegraded":"2023-11-24T06:20:56.451453+0000","last_fullsized":"2023-11-24T06:20:56.451453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442861+0000","last_change":"2023-11-24T06:20:56.442861+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442861+0000","last_undegraded":"2023-11-24T06:20:56.442861+0000","last_fullsized":"2023-11-24T06:20:56.442861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443493+0000","last_change":"2023-11-24T06:20:56.443493+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443493+0000","last_undegraded":"2023-11-24T06:20:56.443493+0000","last_fullsized":"2023-11-24T06:20:56.443493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443677+0000","last_change":"2023-11-24T06:20:56.443677+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443677+0000","last_undegraded":"2023-11-24T06:20:56.443677+0000","last_fullsized":"2023-11-24T06:20:56.443677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438156+0000","last_change":"2023-11-24T06:20:56.438156+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438156+0000","last_undegraded":"2023-11-24T06:20:56.438156+0000","last_fullsized":"2023-11-24T06:20:56.438156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438903+0000","last_change":"2023-11-24T06:20:56.438903+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438903+0000","last_undegraded":"2023-11-24T06:20:56.438903+0000","last_fullsized":"2023-11-24T06:20:56.438903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443185+0000","last_change":"2023-11-24T06:20:56.443185+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443185+0000","last_undegraded":"2023-11-24T06:20:56.443185+0000","last_fullsized":"2023-11-24T06:20:56.443185+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446196+0000","last_change":"2023-11-24T06:20:56.446196+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446196+0000","last_undegraded":"2023-11-24T06:20:56.446196+0000","last_fullsized":"2023-11-24T06:20:56.446196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441252+0000","last_change":"2023-11-24T06:20:56.441252+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441252+0000","last_undegraded":"2023-11-24T06:20:56.441252+0000","last_fullsized":"2023-11-24T06:20:56.441252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438658+0000","last_change":"2023-11-24T06:20:56.438658+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438658+0000","last_undegraded":"2023-11-24T06:20:56.438658+0000","last_fullsized":"2023-11-24T06:20:56.438658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441558+0000","last_change":"2023-11-24T06:20:56.441558+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441558+0000","last_undegraded":"2023-11-24T06:20:56.441558+0000","last_fullsized":"2023-11-24T06:20:56.441558+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445126+0000","last_change":"2023-11-24T06:20:56.445126+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445126+0000","last_undegraded":"2023-11-24T06:20:56.445126+0000","last_fullsized":"2023-11-24T06:20:56.445126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446281+0000","last_change":"2023-11-24T06:20:56.446281+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446281+0000","last_undegraded":"2023-11-24T06:20:56.446281+0000","last_fullsized":"2023-11-24T06:20:56.446281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449025+0000","last_change":"2023-11-24T06:20:56.449025+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449025+0000","last_undegraded":"2023-11-24T06:20:56.449025+0000","last_fullsized":"2023-11-24T06:20:56.449025+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440692+0000","last_change":"2023-11-24T06:20:56.440692+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440692+0000","last_undegraded":"2023-11-24T06:20:56.440692+0000","last_fullsized":"2023-11-24T06:20:56.440692+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.455924+0000","last_change":"2023-11-24T06:20:56.450056+0000","last_active":"2023-11-24T06:20:56.455924+0000","last_peered":"2023-11-24T06:20:56.455924+0000","last_clean":"2023-11-24T06:20:56.455924+0000","last_became_active":"2023-11-24T06:20:56.449760+0000","last_became_peered":"2023-11-24T06:20:56.449760+0000","last_unstale":"2023-11-24T06:20:56.455924+0000","last_undegraded":"2023-11-24T06:20:56.455924+0000","last_fullsized":"2023-11-24T06:20:56.455924+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:42.412869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445988+0000","last_change":"2023-11-24T06:20:56.445988+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445988+0000","last_undegraded":"2023-11-24T06:20:56.445988+0000","last_fullsized":"2023-11-24T06:20:56.445988+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454721+0000","last_change":"2023-11-24T06:20:56.450515+0000","last_active":"2023-11-24T06:20:56.454721+0000","last_peered":"2023-11-24T06:20:56.454721+0000","last_clean":"2023-11-24T06:20:56.454721+0000","last_became_active":"2023-11-24T06:20:56.450219+0000","last_became_peered":"2023-11-24T06:20:56.450219+0000","last_unstale":"2023-11-24T06:20:56.454721+0000","last_undegraded":"2023-11-24T06:20:56.454721+0000","last_fullsized":"2023-11-24T06:20:56.454721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:40:15.263523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.449285+0000","last_change":"2023-11-24T06:20:56.449285+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.449285+0000","last_undegraded":"2023-11-24T06:20:56.449285+0000","last_fullsized":"2023-11-24T06:20:56.449285+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463396+0000","last_change":"2023-11-24T06:20:56.457862+0000","last_active":"2023-11-24T06:20:56.463396+0000","last_peered":"2023-11-24T06:20:56.463396+0000","last_clean":"2023-11-24T06:20:56.463396+0000","last_became_active":"2023-11-24T06:20:56.457485+0000","last_became_peered":"2023-11-24T06:20:56.457485+0000","last_unstale":"2023-11-24T06:20:56.463396+0000","last_undegraded":"2023-11-24T06:20:56.463396+0000","last_fullsized":"2023-11-24T06:20:56.463396+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:44:24.187407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439756+0000","last_change":"2023-11-24T06:20:56.439756+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439756+0000","last_undegraded":"2023-11-24T06:20:56.439756+0000","last_fullsized":"2023-11-24T06:20:56.439756+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461058+0000","last_change":"2023-11-24T06:20:56.456592+0000","last_active":"2023-11-24T06:20:56.461058+0000","last_peered":"2023-11-24T06:20:56.461058+0000","last_clean":"2023-11-24T06:20:56.461058+0000","last_became_active":"2023-11-24T06:20:56.456416+0000","last_became_peered":"2023-11-24T06:20:56.456416+0000","last_unstale":"2023-11-24T06:20:56.461058+0000","last_undegraded":"2023-11-24T06:20:56.461058+0000","last_fullsized":"2023-11-24T06:20:56.461058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:19:08.950567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444793+0000","last_change":"2023-11-24T06:20:56.444793+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444793+0000","last_undegraded":"2023-11-24T06:20:56.444793+0000","last_fullsized":"2023-11-24T06:20:56.444793+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.455514+0000","last_change":"2023-11-24T06:20:56.452696+0000","last_active":"2023-11-24T06:20:56.455514+0000","last_peered":"2023-11-24T06:20:56.455514+0000","last_clean":"2023-11-24T06:20:56.455514+0000","last_became_active":"2023-11-24T06:20:56.452226+0000","last_became_peered":"2023-11-24T06:20:56.452226+0000","last_unstale":"2023-11-24T06:20:56.455514+0000","last_undegraded":"2023-11-24T06:20:56.455514+0000","last_fullsized":"2023-11-24T06:20:56.455514+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:59:01.228879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438821+0000","last_change":"2023-11-24T06:20:56.438821+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438821+0000","last_undegraded":"2023-11-24T06:20:56.438821+0000","last_fullsized":"2023-11-24T06:20:56.438821+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463517+0000","last_change":"2023-11-24T06:20:56.459294+0000","last_active":"2023-11-24T06:20:56.463517+0000","last_peered":"2023-11-24T06:20:56.463517+0000","last_clean":"2023-11-24T06:20:56.463517+0000","last_became_active":"2023-11-24T06:20:56.458979+0000","last_became_peered":"2023-11-24T06:20:56.458979+0000","last_unstale":"2023-11-24T06:20:56.463517+0000","last_undegraded":"2023-11-24T06:20:56.463517+0000","last_fullsized":"2023-11-24T06:20:56.463517+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:26.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446512+0000","last_change":"2023-11-24T06:20:56.446512+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446512+0000","last_undegraded":"2023-11-24T06:20:56.446512+0000","last_fullsized":"2023-11-24T06:20:56.446512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454765+0000","last_change":"2023-11-24T06:20:56.451454+0000","last_active":"2023-11-24T06:20:56.454765+0000","last_peered":"2023-11-24T06:20:56.454765+0000","last_clean":"2023-11-24T06:20:56.454765+0000","last_became_active":"2023-11-24T06:20:56.451064+0000","last_became_peered":"2023-11-24T06:20:56.451064+0000","last_unstale":"2023-11-24T06:20:56.454765+0000","last_undegraded":"2023-11-24T06:20:56.454765+0000","last_fullsized":"2023-11-24T06:20:56.454765+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442438+0000","last_change":"2023-11-24T06:20:56.442438+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442438+0000","last_undegraded":"2023-11-24T06:20:56.442438+0000","last_fullsized":"2023-11-24T06:20:56.442438+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454770+0000","last_change":"2023-11-24T06:20:56.449997+0000","last_active":"2023-11-24T06:20:56.454770+0000","last_peered":"2023-11-24T06:20:56.454770+0000","last_clean":"2023-11-24T06:20:56.454770+0000","last_became_active":"2023-11-24T06:20:56.449727+0000","last_became_peered":"2023-11-24T06:20:56.449727+0000","last_unstale":"2023-11-24T06:20:56.454770+0000","last_undegraded":"2023-11-24T06:20:56.454770+0000","last_fullsized":"2023-11-24T06:20:56.454770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:13.838572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445798+0000","last_change":"2023-11-24T06:20:56.445798+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445798+0000","last_undegraded":"2023-11-24T06:20:56.445798+0000","last_fullsized":"2023-11-24T06:20:56.445798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452073+0000","last_change":"2023-11-24T06:20:56.444835+0000","last_active":"2023-11-24T06:20:56.452073+0000","last_peered":"2023-11-24T06:20:56.452073+0000","last_clean":"2023-11-24T06:20:56.452073+0000","last_became_active":"2023-11-24T06:20:56.444547+0000","last_became_peered":"2023-11-24T06:20:56.444547+0000","last_unstale":"2023-11-24T06:20:56.452073+0000","last_undegraded":"2023-11-24T06:20:56.452073+0000","last_fullsized":"2023-11-24T06:20:56.452073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:12.584029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441093+0000","last_change":"2023-11-24T06:20:56.441093+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441093+0000","last_undegraded":"2023-11-24T06:20:56.441093+0000","last_fullsized":"2023-11-24T06:20:56.441093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458584+0000","last_change":"2023-11-24T06:20:56.455466+0000","last_active":"2023-11-24T06:20:56.458584+0000","last_peered":"2023-11-24T06:20:56.458584+0000","last_clean":"2023-11-24T06:20:56.458584+0000","last_became_active":"2023-11-24T06:20:56.455137+0000","last_became_peered":"2023-11-24T06:20:56.455137+0000","last_unstale":"2023-11-24T06:20:56.458584+0000","last_undegraded":"2023-11-24T06:20:56.458584+0000","last_fullsized":"2023-11-24T06:20:56.458584+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:25:31.853530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446530+0000","last_change":"2023-11-24T06:20:56.446530+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446530+0000","last_undegraded":"2023-11-24T06:20:56.446530+0000","last_fullsized":"2023-11-24T06:20:56.446530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.456617+0000","last_change":"2023-11-24T06:20:56.454452+0000","last_active":"2023-11-24T06:20:56.456617+0000","last_peered":"2023-11-24T06:20:56.456617+0000","last_clean":"2023-11-24T06:20:56.456617+0000","last_became_active":"2023-11-24T06:20:56.454143+0000","last_became_peered":"2023-11-24T06:20:56.454143+0000","last_unstale":"2023-11-24T06:20:56.456617+0000","last_undegraded":"2023-11-24T06:20:56.456617+0000","last_fullsized":"2023-11-24T06:20:56.456617+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:31.618289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445684+0000","last_change":"2023-11-24T06:20:56.445684+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445684+0000","last_undegraded":"2023-11-24T06:20:56.445684+0000","last_fullsized":"2023-11-24T06:20:56.445684+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454599+0000","last_change":"2023-11-24T06:20:56.450022+0000","last_active":"2023-11-24T06:20:56.454599+0000","last_peered":"2023-11-24T06:20:56.454599+0000","last_clean":"2023-11-24T06:20:56.454599+0000","last_became_active":"2023-11-24T06:20:56.449589+0000","last_became_peered":"2023-11-24T06:20:56.449589+0000","last_unstale":"2023-11-24T06:20:56.454599+0000","last_undegraded":"2023-11-24T06:20:56.454599+0000","last_fullsized":"2023-11-24T06:20:56.454599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.438775+0000","last_change":"2023-11-24T06:20:56.438775+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.438775+0000","last_undegraded":"2023-11-24T06:20:56.438775+0000","last_fullsized":"2023-11-24T06:20:56.438775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452082+0000","last_change":"2023-11-24T06:20:56.448978+0000","last_active":"2023-11-24T06:20:56.452082+0000","last_peered":"2023-11-24T06:20:56.452082+0000","last_clean":"2023-11-24T06:20:56.452082+0000","last_became_active":"2023-11-24T06:20:56.448657+0000","last_became_peered":"2023-11-24T06:20:56.448657+0000","last_unstale":"2023-11-24T06:20:56.452082+0000","last_undegraded":"2023-11-24T06:20:56.452082+0000","last_fullsized":"2023-11-24T06:20:56.452082+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:57:12.584029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.443962+0000","last_change":"2023-11-24T06:20:56.443962+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.443962+0000","last_undegraded":"2023-11-24T06:20:56.443962+0000","last_fullsized":"2023-11-24T06:20:56.443962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458629+0000","last_change":"2023-11-24T06:20:56.455610+0000","last_active":"2023-11-24T06:20:56.458629+0000","last_peered":"2023-11-24T06:20:56.458629+0000","last_clean":"2023-11-24T06:20:56.458629+0000","last_became_active":"2023-11-24T06:20:56.455289+0000","last_became_peered":"2023-11-24T06:20:56.455289+0000","last_unstale":"2023-11-24T06:20:56.458629+0000","last_undegraded":"2023-11-24T06:20:56.458629+0000","last_fullsized":"2023-11-24T06:20:56.458629+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:17.452542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442464+0000","last_change":"2023-11-24T06:20:56.442464+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442464+0000","last_undegraded":"2023-11-24T06:20:56.442464+0000","last_fullsized":"2023-11-24T06:20:56.442464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458511+0000","last_change":"2023-11-24T06:20:56.455905+0000","last_active":"2023-11-24T06:20:56.458511+0000","last_peered":"2023-11-24T06:20:56.458511+0000","last_clean":"2023-11-24T06:20:56.458511+0000","last_became_active":"2023-11-24T06:20:56.455373+0000","last_became_peered":"2023-11-24T06:20:56.455373+0000","last_unstale":"2023-11-24T06:20:56.458511+0000","last_undegraded":"2023-11-24T06:20:56.458511+0000","last_fullsized":"2023-11-24T06:20:56.458511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446537+0000","last_change":"2023-11-24T06:20:56.446537+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446537+0000","last_undegraded":"2023-11-24T06:20:56.446537+0000","last_fullsized":"2023-11-24T06:20:56.446537+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452266+0000","last_change":"2023-11-24T06:20:56.445784+0000","last_active":"2023-11-24T06:20:56.452266+0000","last_peered":"2023-11-24T06:20:56.452266+0000","last_clean":"2023-11-24T06:20:56.452266+0000","last_became_active":"2023-11-24T06:20:56.445468+0000","last_became_peered":"2023-11-24T06:20:56.445468+0000","last_unstale":"2023-11-24T06:20:56.452266+0000","last_undegraded":"2023-11-24T06:20:56.452266+0000","last_fullsized":"2023-11-24T06:20:56.452266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442896+0000","last_change":"2023-11-24T06:20:56.442896+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442896+0000","last_undegraded":"2023-11-24T06:20:56.442896+0000","last_fullsized":"2023-11-24T06:20:56.442896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461188+0000","last_change":"2023-11-24T06:20:56.458379+0000","last_active":"2023-11-24T06:20:56.461188+0000","last_peered":"2023-11-24T06:20:56.461188+0000","last_clean":"2023-11-24T06:20:56.461188+0000","last_became_active":"2023-11-24T06:20:56.457956+0000","last_became_peered":"2023-11-24T06:20:56.457956+0000","last_unstale":"2023-11-24T06:20:56.461188+0000","last_undegraded":"2023-11-24T06:20:56.461188+0000","last_fullsized":"2023-11-24T06:20:56.461188+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:20.310219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440565+0000","last_change":"2023-11-24T06:20:56.440565+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440565+0000","last_undegraded":"2023-11-24T06:20:56.440565+0000","last_fullsized":"2023-11-24T06:20:56.440565+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463426+0000","last_change":"2023-11-24T06:20:56.459307+0000","last_active":"2023-11-24T06:20:56.463426+0000","last_peered":"2023-11-24T06:20:56.463426+0000","last_clean":"2023-11-24T06:20:56.463426+0000","last_became_active":"2023-11-24T06:20:56.459077+0000","last_became_peered":"2023-11-24T06:20:56.459077+0000","last_unstale":"2023-11-24T06:20:56.463426+0000","last_undegraded":"2023-11-24T06:20:56.463426+0000","last_fullsized":"2023-11-24T06:20:56.463426+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:07.713106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445945+0000","last_change":"2023-11-24T06:20:56.445945+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445945+0000","last_undegraded":"2023-11-24T06:20:56.445945+0000","last_fullsized":"2023-11-24T06:20:56.445945+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461147+0000","last_change":"2023-11-24T06:20:56.457538+0000","last_active":"2023-11-24T06:20:56.461147+0000","last_peered":"2023-11-24T06:20:56.461147+0000","last_clean":"2023-11-24T06:20:56.461147+0000","last_became_active":"2023-11-24T06:20:56.457247+0000","last_became_peered":"2023-11-24T06:20:56.457247+0000","last_unstale":"2023-11-24T06:20:56.461147+0000","last_undegraded":"2023-11-24T06:20:56.461147+0000","last_fullsized":"2023-11-24T06:20:56.461147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:50.332536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445239+0000","last_change":"2023-11-24T06:20:56.445239+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445239+0000","last_undegraded":"2023-11-24T06:20:56.445239+0000","last_fullsized":"2023-11-24T06:20:56.445239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.457746+0000","last_change":"2023-11-24T06:20:56.453032+0000","last_active":"2023-11-24T06:20:56.457746+0000","last_peered":"2023-11-24T06:20:56.457746+0000","last_clean":"2023-11-24T06:20:56.457746+0000","last_became_active":"2023-11-24T06:20:56.452748+0000","last_became_peered":"2023-11-24T06:20:56.452748+0000","last_unstale":"2023-11-24T06:20:56.457746+0000","last_undegraded":"2023-11-24T06:20:56.457746+0000","last_fullsized":"2023-11-24T06:20:56.457746+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:37.569663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441010+0000","last_change":"2023-11-24T06:20:56.441010+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441010+0000","last_undegraded":"2023-11-24T06:20:56.441010+0000","last_fullsized":"2023-11-24T06:20:56.441010+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.463502+0000","last_change":"2023-11-24T06:20:56.459215+0000","last_active":"2023-11-24T06:20:56.463502+0000","last_peered":"2023-11-24T06:20:56.463502+0000","last_clean":"2023-11-24T06:20:56.463502+0000","last_became_active":"2023-11-24T06:20:56.458937+0000","last_became_peered":"2023-11-24T06:20:56.458937+0000","last_unstale":"2023-11-24T06:20:56.463502+0000","last_undegraded":"2023-11-24T06:20:56.463502+0000","last_fullsized":"2023-11-24T06:20:56.463502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:03:06.709201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440845+0000","last_change":"2023-11-24T06:20:56.440845+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440845+0000","last_undegraded":"2023-11-24T06:20:56.440845+0000","last_fullsized":"2023-11-24T06:20:56.440845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458490+0000","last_change":"2023-11-24T06:20:56.455809+0000","last_active":"2023-11-24T06:20:56.458490+0000","last_peered":"2023-11-24T06:20:56.458490+0000","last_clean":"2023-11-24T06:20:56.458490+0000","last_became_active":"2023-11-24T06:20:56.455461+0000","last_became_peered":"2023-11-24T06:20:56.455461+0000","last_unstale":"2023-11-24T06:20:56.458490+0000","last_undegraded":"2023-11-24T06:20:56.458490+0000","last_fullsized":"2023-11-24T06:20:56.458490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.441040+0000","last_change":"2023-11-24T06:20:56.441040+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.441040+0000","last_undegraded":"2023-11-24T06:20:56.441040+0000","last_fullsized":"2023-11-24T06:20:56.441040+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.454779+0000","last_change":"2023-11-24T06:20:56.450795+0000","last_active":"2023-11-24T06:20:56.454779+0000","last_peered":"2023-11-24T06:20:56.454779+0000","last_clean":"2023-11-24T06:20:56.454779+0000","last_became_active":"2023-11-24T06:20:56.450111+0000","last_became_peered":"2023-11-24T06:20:56.450111+0000","last_unstale":"2023-11-24T06:20:56.454779+0000","last_undegraded":"2023-11-24T06:20:56.454779+0000","last_fullsized":"2023-11-24T06:20:56.454779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.442840+0000","last_change":"2023-11-24T06:20:56.442840+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.442840+0000","last_undegraded":"2023-11-24T06:20:56.442840+0000","last_fullsized":"2023-11-24T06:20:56.442840+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461173+0000","last_change":"2023-11-24T06:20:56.458175+0000","last_active":"2023-11-24T06:20:56.461173+0000","last_peered":"2023-11-24T06:20:56.461173+0000","last_clean":"2023-11-24T06:20:56.461173+0000","last_became_active":"2023-11-24T06:20:56.457570+0000","last_became_peered":"2023-11-24T06:20:56.457570+0000","last_unstale":"2023-11-24T06:20:56.461173+0000","last_undegraded":"2023-11-24T06:20:56.461173+0000","last_fullsized":"2023-11-24T06:20:56.461173+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:05.207780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444484+0000","last_change":"2023-11-24T06:20:56.444484+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444484+0000","last_undegraded":"2023-11-24T06:20:56.444484+0000","last_fullsized":"2023-11-24T06:20:56.444484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458402+0000","last_change":"2023-11-24T06:20:56.454563+0000","last_active":"2023-11-24T06:20:56.458402+0000","last_peered":"2023-11-24T06:20:56.458402+0000","last_clean":"2023-11-24T06:20:56.458402+0000","last_became_active":"2023-11-24T06:20:56.454014+0000","last_became_peered":"2023-11-24T06:20:56.454014+0000","last_unstale":"2023-11-24T06:20:56.458402+0000","last_undegraded":"2023-11-24T06:20:56.458402+0000","last_fullsized":"2023-11-24T06:20:56.458402+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:22:26.548969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439927+0000","last_change":"2023-11-24T06:20:56.439927+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439927+0000","last_undegraded":"2023-11-24T06:20:56.439927+0000","last_fullsized":"2023-11-24T06:20:56.439927+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458720+0000","last_change":"2023-11-24T06:20:56.455444+0000","last_active":"2023-11-24T06:20:56.458720+0000","last_peered":"2023-11-24T06:20:56.458720+0000","last_clean":"2023-11-24T06:20:56.458720+0000","last_became_active":"2023-11-24T06:20:56.455167+0000","last_became_peered":"2023-11-24T06:20:56.455167+0000","last_unstale":"2023-11-24T06:20:56.458720+0000","last_undegraded":"2023-11-24T06:20:56.458720+0000","last_fullsized":"2023-11-24T06:20:56.458720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:59.246341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.439301+0000","last_change":"2023-11-24T06:20:56.439301+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.439301+0000","last_undegraded":"2023-11-24T06:20:56.439301+0000","last_fullsized":"2023-11-24T06:20:56.439301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.457669+0000","last_change":"2023-11-24T06:20:56.453405+0000","last_active":"2023-11-24T06:20:56.457669+0000","last_peered":"2023-11-24T06:20:56.457669+0000","last_clean":"2023-11-24T06:20:56.457669+0000","last_became_active":"2023-11-24T06:20:56.453077+0000","last_became_peered":"2023-11-24T06:20:56.453077+0000","last_unstale":"2023-11-24T06:20:56.457669+0000","last_undegraded":"2023-11-24T06:20:56.457669+0000","last_fullsized":"2023-11-24T06:20:56.457669+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444847+0000","last_change":"2023-11-24T06:20:56.444847+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444847+0000","last_undegraded":"2023-11-24T06:20:56.444847+0000","last_fullsized":"2023-11-24T06:20:56.444847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.458271+0000","last_change":"2023-11-24T06:20:56.455600+0000","last_active":"2023-11-24T06:20:56.458271+0000","last_peered":"2023-11-24T06:20:56.458271+0000","last_clean":"2023-11-24T06:20:56.458271+0000","last_became_active":"2023-11-24T06:20:56.455299+0000","last_became_peered":"2023-11-24T06:20:56.455299+0000","last_unstale":"2023-11-24T06:20:56.458271+0000","last_undegraded":"2023-11-24T06:20:56.458271+0000","last_fullsized":"2023-11-24T06:20:56.458271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:41:00.663712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.446278+0000","last_change":"2023-11-24T06:20:56.446278+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.446278+0000","last_undegraded":"2023-11-24T06:20:56.446278+0000","last_fullsized":"2023-11-24T06:20:56.446278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.456003+0000","last_change":"2023-11-24T06:20:56.453053+0000","last_active":"2023-11-24T06:20:56.456003+0000","last_peered":"2023-11-24T06:20:56.456003+0000","last_clean":"2023-11-24T06:20:56.456003+0000","last_became_active":"2023-11-24T06:20:56.452778+0000","last_became_peered":"2023-11-24T06:20:56.452778+0000","last_unstale":"2023-11-24T06:20:56.456003+0000","last_undegraded":"2023-11-24T06:20:56.456003+0000","last_fullsized":"2023-11-24T06:20:56.456003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:54.169232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.444952+0000","last_change":"2023-11-24T06:20:56.444952+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.444952+0000","last_undegraded":"2023-11-24T06:20:56.444952+0000","last_fullsized":"2023-11-24T06:20:56.444952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.452274+0000","last_change":"2023-11-24T06:20:56.445884+0000","last_active":"2023-11-24T06:20:56.452274+0000","last_peered":"2023-11-24T06:20:56.452274+0000","last_clean":"2023-11-24T06:20:56.452274+0000","last_became_active":"2023-11-24T06:20:56.445600+0000","last_became_peered":"2023-11-24T06:20:56.445600+0000","last_unstale":"2023-11-24T06:20:56.452274+0000","last_undegraded":"2023-11-24T06:20:56.452274+0000","last_fullsized":"2023-11-24T06:20:56.452274+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.445794+0000","last_change":"2023-11-24T06:20:56.445794+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.445794+0000","last_undegraded":"2023-11-24T06:20:56.445794+0000","last_fullsized":"2023-11-24T06:20:56.445794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461106+0000","last_change":"2023-11-24T06:20:56.456769+0000","last_active":"2023-11-24T06:20:56.461106+0000","last_peered":"2023-11-24T06:20:56.461106+0000","last_clean":"2023-11-24T06:20:56.461106+0000","last_became_active":"2023-11-24T06:20:56.456558+0000","last_became_peered":"2023-11-24T06:20:56.456558+0000","last_unstale":"2023-11-24T06:20:56.461106+0000","last_undegraded":"2023-11-24T06:20:56.461106+0000","last_fullsized":"2023-11-24T06:20:56.461106+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:37:47.748375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":62,"state":"creating+peering","last_fresh":"2023-11-24T06:20:56.440874+0000","last_change":"2023-11-24T06:20:56.440874+0000","last_active":"2023-11-24T06:20:56.425380+0000","last_peered":"2023-11-24T06:20:56.425380+0000","last_clean":"2023-11-24T06:20:56.425380+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:20:56.440874+0000","last_undegraded":"2023-11-24T06:20:56.440874+0000","last_fullsized":"2023-11-24T06:20:56.440874+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-24T06:20:56.461112+0000","last_change":"2023-11-24T06:20:56.456842+0000","last_active":"2023-11-24T06:20:56.461112+0000","last_peered":"2023-11-24T06:20:56.461112+0000","last_clean":"2023-11-24T06:20:56.461112+0000","last_became_active":"2023-11-24T06:20:56.456506+0000","last_became_peered":"2023-11-24T06:20:56.456506+0000","last_unstale":"2023-11-24T06:20:56.461112+0000","last_undegraded":"2023-11-24T06:20:56.461112+0000","last_fullsized":"2023-11-24T06:20:56.461112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:03:05.191510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":207,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-11-24T06:20:55.430113+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:55.430113+0000","last_peered":"2023-11-24T06:20:55.430113+0000","last_clean":"2023-11-24T06:20:55.430113+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:55.430113+0000","last_undegraded":"2023-11-24T06:20:55.430113+0000","last_fullsized":"2023-11-24T06:20:55.430113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28340,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343500,"statfs":{"total":96636764160,"available":96607744000,"internally_reserved":0,"allocated":1335296,"data_stored":904636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59699999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.505},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.628}]}]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.308},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.018}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.625}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.439}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.5}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.60899999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.65600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":753664,"data_stored":320543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.251},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.32800000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.37},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.314}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.54000000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.70299999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.73399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28328,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343512,"statfs":{"total":96636764160,"available":96607756288,"internally_reserved":0,"allocated":1335296,"data_stored":904636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.317},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.625}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.59499999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.251}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.38200000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.54100000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.624}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.26500000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.247},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32500000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.432},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.47899999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.45300000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58699999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.36899999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.40300000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.5}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":314268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.505},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.68200000000000005}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.35899999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41699999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:06.125 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:06.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.258+0000 7f26fb55d700 1 Processor -- start 2023-11-24T06:21:06.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.258+0000 7f26fb55d700 1 -- start start 2023-11-24T06:21:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26fb55d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406c710 0x7f26f40b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26fb55d700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f40ae430 con 0x7f26f40af4d0 2023-11-24T06:21:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26fb55d700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f40ae710 con 0x7f26f415e080 2023-11-24T06:21:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26fb55d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40af0b0 con 0x7f26f406c710 2023-11-24T06:21:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26f9afa700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f26f40af4d0 0x7f26f406e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42556/0 (socket says 172.21.15.89:42556) 2023-11-24T06:21:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26f9afa700 1 -- 172.21.15.89:0/2902030538 learned_addr learned my addr 172.21.15.89:0/2902030538 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26f8af8700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406c710 0x7f26f40b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239721395 0 0) 0x7f26f40ae430 con 0x7f26f40af4d0 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.259+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e00035b0 con 0x7f26f40af4d0 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3039719094 0 0) 0x7f26e00035b0 con 0x7f26f40af4d0 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406c710 msgr2=0x7f26f40b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406c710 0x7f26f40b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 >> v1:172.21.15.123:6789/0 conn(0x7f26f415e080 legacy=0x7f26f406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f40ae9f0 con 0x7f26f40af4d0 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f26ec0019f0 con 0x7f26f40af4d0 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 >> v1:172.21.15.89:6789/0 conn(0x7f26f40af4d0 legacy=0x7f26f406e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e0003a30 0x7f26e0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26e00048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406c710 0x7f26e0005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40af0b0 con 0x7f26e00043e0 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e000ba10 con 0x7f26e0003a30 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e000bbd0 con 0x7f26f406c710 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f8af8700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26e00048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f92f9700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e0003a30 0x7f26e0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f9afa700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406c710 0x7f26e0005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f8af8700 1 -- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406c710 msgr2=0x7f26e0005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f8af8700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406c710 0x7f26e0005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f8af8700 1 -- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e0003a30 msgr2=0x7f26e0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f8af8700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e0003a30 0x7f26e0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f8af8700 1 -- 172.21.15.89:0/2902030538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00035b0 con 0x7f26e00043e0 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26f8af8700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26e00048b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009b60 tx=0x7f26e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfad116637f43ff server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e400e6c0 con 0x7f26e00043e0 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e400e820 con 0x7f26e00043e0 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.260+0000 7f26ebfff700 1 -- 172.21.15.89:0/2902030538 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e400eb90 con 0x7f26e00043e0 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.261+0000 7f26fb55d700 1 -- 172.21.15.89:0/2902030538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 msgr2=0x7f26e00048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.261+0000 7f26fb55d700 1 --2- 172.21.15.89:0/2902030538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26e00048b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009b60 tx=0x7f26e0005880 comp rx=0 tx=0).stop 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.261+0000 7f26f9afa700 1 -- 172.21.15.89:0/2902030538 reap_dead start 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.261+0000 7f26fb55d700 1 -- 172.21.15.89:0/2902030538 shutdown_connections 2023-11-24T06:21:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.261+0000 7f26fb55d700 1 -- 172.21.15.89:0/2902030538 >> 172.21.15.89:0/2902030538 conn(0x7f26f4071b90 msgr2=0x7f26f4071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:06.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.261+0000 7f26fb55d700 1 -- 172.21.15.89:0/2902030538 shutdown_connections 2023-11-24T06:21:06.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.261+0000 7f26fb55d700 1 -- 172.21.15.89:0/2902030538 wait complete. 2023-11-24T06:21:06.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.262+0000 7f26fb55d700 1 Processor -- start 2023-11-24T06:21:06.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.262+0000 7f26fb55d700 1 -- start start 2023-11-24T06:21:06.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26fb55d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26f415ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26fb55d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f415f390 0x7f26f4156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26fb55d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f4157500 0x7f26f41420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26fb55d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4157990 con 0x7f26e00043e0 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26fb55d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4141c20 con 0x7f26f415f390 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26fb55d700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4141010 con 0x7f26f4157500 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26f92f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26f415ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26f92f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26f415ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39060/0 (socket says 172.21.15.89:39060) 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26f92f9700 1 -- 172.21.15.89:0/3046204810 learned_addr learned my addr 172.21.15.89:0/3046204810 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26f92f9700 1 -- 172.21.15.89:0/3046204810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f4157500 msgr2=0x7f26f41420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26f92f9700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f4157500 0x7f26f41420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26f92f9700 1 -- 172.21.15.89:0/3046204810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f415f390 msgr2=0x7f26f4156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26f92f9700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f415f390 0x7f26f4156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.263+0000 7f26f92f9700 1 -- 172.21.15.89:0/3046204810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f41412f0 con 0x7f26e00043e0 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.264+0000 7f26f92f9700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26f415ee50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f26f000f770 tx=0x7f26f000fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.264+0000 7f26e9ffb700 1 -- 172.21.15.89:0/3046204810 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0001460 con 0x7f26e00043e0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.264+0000 7f26e9ffb700 1 -- 172.21.15.89:0/3046204810 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f00052e0 con 0x7f26e00043e0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.264+0000 7f26e9ffb700 1 -- 172.21.15.89:0/3046204810 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0003680 con 0x7f26e00043e0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.264+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f41415d0 con 0x7f26e00043e0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.264+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f4140c50 con 0x7f26e00043e0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.265+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f40753f0 con 0x7f26e00043e0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.265+0000 7f26e9ffb700 1 -- 172.21.15.89:0/3046204810 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f26f00037e0 con 0x7f26e00043e0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.266+0000 7f26e9ffb700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f26d806af60 0x7f26d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.266+0000 7f26f8af8700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f26d806af60 0x7f26d806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.266+0000 7f26e9ffb700 1 -- 172.21.15.89:0/3046204810 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26f00894f0 con 0x7f26e00043e0 2023-11-24T06:21:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.266+0000 7f26f8af8700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f26d806af60 0x7f26d806d420 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f26f415f790 tx=0x7f26e4013000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:06.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.270+0000 7f26e9ffb700 1 -- 172.21.15.89:0/3046204810 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26f00545b0 con 0x7f26e00043e0 2023-11-24T06:21:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.456+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26f40b05a0 con 0x7f26d806af60 2023-11-24T06:21:06.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.464+0000 7f26e9ffb700 1 -- 172.21.15.89:0/3046204810 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289439 (secure 0 0 0) 0x7f26e400fa20 con 0x7f26d806af60 2023-11-24T06:21:06.469 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f26d806af60 msgr2=0x7f26d806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f26d806af60 0x7f26d806d420 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f26f415f790 tx=0x7f26e4013000 comp rx=0 tx=0).stop 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 msgr2=0x7f26f415ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26f415ee50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f26f000f770 tx=0x7f26f000fb30 comp rx=0 tx=0).stop 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 shutdown_connections 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f4157500 0x7f26f41420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e00043e0 0x7f26f415ee50 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f26d806af60 0x7f26d806d420 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 --2- 172.21.15.89:0/3046204810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f415f390 0x7f26f4156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 >> 172.21.15.89:0/3046204810 conn(0x7f26f4071b90 msgr2=0x7f26f4156540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:06.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 shutdown_connections 2023-11-24T06:21:06.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.467+0000 7f26fb55d700 1 -- 172.21.15.89:0/3046204810 wait complete. 2023-11-24T06:21:06.473 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:06.481 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2023-11-24T06:21:05.746437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223480,"kb_used_data":7304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751240,"statfs":{"total":773094113280,"available":772865269760,"internally_reserved":0,"allocated":7479296,"data_stored":3854500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.311547"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441519+0000","last_change":"2023-11-24T06:20:57.448900+0000","last_active":"2023-11-24T06:20:58.441519+0000","last_peered":"2023-11-24T06:20:58.441519+0000","last_clean":"2023-11-24T06:20:58.441519+0000","last_became_active":"2023-11-24T06:20:57.448089+0000","last_became_peered":"2023-11-24T06:20:57.448089+0000","last_unstale":"2023-11-24T06:20:58.441519+0000","last_undegraded":"2023-11-24T06:20:58.441519+0000","last_fullsized":"2023-11-24T06:20:58.441519+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:20:35.543303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637228+0000","last_change":"2023-11-24T06:20:57.467654+0000","last_active":"2023-11-24T06:20:58.637228+0000","last_peered":"2023-11-24T06:20:58.637228+0000","last_clean":"2023-11-24T06:20:58.637228+0000","last_became_active":"2023-11-24T06:20:57.467394+0000","last_became_peered":"2023-11-24T06:20:57.467394+0000","last_unstale":"2023-11-24T06:20:58.637228+0000","last_undegraded":"2023-11-24T06:20:58.637228+0000","last_fullsized":"2023-11-24T06:20:58.637228+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:29.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.437095+0000","last_change":"2023-11-24T06:20:57.461202+0000","last_active":"2023-11-24T06:20:58.437095+0000","last_peered":"2023-11-24T06:20:58.437095+0000","last_clean":"2023-11-24T06:20:58.437095+0000","last_became_active":"2023-11-24T06:20:57.460820+0000","last_became_peered":"2023-11-24T06:20:57.460820+0000","last_unstale":"2023-11-24T06:20:58.437095+0000","last_undegraded":"2023-11-24T06:20:58.437095+0000","last_fullsized":"2023-11-24T06:20:58.437095+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:58:46.958012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.635629+0000","last_change":"2023-11-24T06:20:57.455694+0000","last_active":"2023-11-24T06:20:58.635629+0000","last_peered":"2023-11-24T06:20:58.635629+0000","last_clean":"2023-11-24T06:20:58.635629+0000","last_became_active":"2023-11-24T06:20:57.455404+0000","last_became_peered":"2023-11-24T06:20:57.455404+0000","last_unstale":"2023-11-24T06:20:58.635629+0000","last_undegraded":"2023-11-24T06:20:58.635629+0000","last_fullsized":"2023-11-24T06:20:58.635629+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:40:16.265148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636229+0000","last_change":"2023-11-24T06:20:57.448860+0000","last_active":"2023-11-24T06:20:58.636229+0000","last_peered":"2023-11-24T06:20:58.636229+0000","last_clean":"2023-11-24T06:20:58.636229+0000","last_became_active":"2023-11-24T06:20:57.448151+0000","last_became_peered":"2023-11-24T06:20:57.448151+0000","last_unstale":"2023-11-24T06:20:58.636229+0000","last_undegraded":"2023-11-24T06:20:58.636229+0000","last_fullsized":"2023-11-24T06:20:58.636229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:38.962455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636877+0000","last_change":"2023-11-24T06:20:57.461379+0000","last_active":"2023-11-24T06:20:58.636877+0000","last_peered":"2023-11-24T06:20:58.636877+0000","last_clean":"2023-11-24T06:20:58.636877+0000","last_became_active":"2023-11-24T06:20:57.461086+0000","last_became_peered":"2023-11-24T06:20:57.461086+0000","last_unstale":"2023-11-24T06:20:58.636877+0000","last_undegraded":"2023-11-24T06:20:58.636877+0000","last_fullsized":"2023-11-24T06:20:58.636877+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:28:53.511548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636145+0000","last_change":"2023-11-24T06:20:57.460439+0000","last_active":"2023-11-24T06:20:58.636145+0000","last_peered":"2023-11-24T06:20:58.636145+0000","last_clean":"2023-11-24T06:20:58.636145+0000","last_became_active":"2023-11-24T06:20:57.460187+0000","last_became_peered":"2023-11-24T06:20:57.460187+0000","last_unstale":"2023-11-24T06:20:58.636145+0000","last_undegraded":"2023-11-24T06:20:58.636145+0000","last_fullsized":"2023-11-24T06:20:58.636145+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:34.947730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636704+0000","last_change":"2023-11-24T06:20:57.455124+0000","last_active":"2023-11-24T06:20:58.636704+0000","last_peered":"2023-11-24T06:20:58.636704+0000","last_clean":"2023-11-24T06:20:58.636704+0000","last_became_active":"2023-11-24T06:20:57.453361+0000","last_became_peered":"2023-11-24T06:20:57.453361+0000","last_unstale":"2023-11-24T06:20:58.636704+0000","last_undegraded":"2023-11-24T06:20:58.636704+0000","last_fullsized":"2023-11-24T06:20:58.636704+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:10:11.292444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636462+0000","last_change":"2023-11-24T06:20:57.462579+0000","last_active":"2023-11-24T06:20:58.636462+0000","last_peered":"2023-11-24T06:20:58.636462+0000","last_clean":"2023-11-24T06:20:58.636462+0000","last_became_active":"2023-11-24T06:20:57.462278+0000","last_became_peered":"2023-11-24T06:20:57.462278+0000","last_unstale":"2023-11-24T06:20:58.636462+0000","last_undegraded":"2023-11-24T06:20:58.636462+0000","last_fullsized":"2023-11-24T06:20:58.636462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:59.369778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.638055+0000","last_change":"2023-11-24T06:20:57.458316+0000","last_active":"2023-11-24T06:20:58.638055+0000","last_peered":"2023-11-24T06:20:58.638055+0000","last_clean":"2023-11-24T06:20:58.638055+0000","last_became_active":"2023-11-24T06:20:57.457989+0000","last_became_peered":"2023-11-24T06:20:57.457989+0000","last_unstale":"2023-11-24T06:20:58.638055+0000","last_undegraded":"2023-11-24T06:20:58.638055+0000","last_fullsized":"2023-11-24T06:20:58.638055+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:38.267765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441091+0000","last_change":"2023-11-24T06:20:57.461920+0000","last_active":"2023-11-24T06:20:58.441091+0000","last_peered":"2023-11-24T06:20:58.441091+0000","last_clean":"2023-11-24T06:20:58.441091+0000","last_became_active":"2023-11-24T06:20:57.461485+0000","last_became_peered":"2023-11-24T06:20:57.461485+0000","last_unstale":"2023-11-24T06:20:58.441091+0000","last_undegraded":"2023-11-24T06:20:58.441091+0000","last_fullsized":"2023-11-24T06:20:58.441091+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:55:57.775216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.635776+0000","last_change":"2023-11-24T06:20:57.449244+0000","last_active":"2023-11-24T06:20:58.635776+0000","last_peered":"2023-11-24T06:20:58.635776+0000","last_clean":"2023-11-24T06:20:58.635776+0000","last_became_active":"2023-11-24T06:20:57.448809+0000","last_became_peered":"2023-11-24T06:20:57.448809+0000","last_unstale":"2023-11-24T06:20:58.635776+0000","last_undegraded":"2023-11-24T06:20:58.635776+0000","last_fullsized":"2023-11-24T06:20:58.635776+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:22:27.550594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.439809+0000","last_change":"2023-11-24T06:20:57.461319+0000","last_active":"2023-11-24T06:20:58.439809+0000","last_peered":"2023-11-24T06:20:58.439809+0000","last_clean":"2023-11-24T06:20:58.439809+0000","last_became_active":"2023-11-24T06:20:57.461074+0000","last_became_peered":"2023-11-24T06:20:57.461074+0000","last_unstale":"2023-11-24T06:20:58.439809+0000","last_undegraded":"2023-11-24T06:20:58.439809+0000","last_fullsized":"2023-11-24T06:20:58.439809+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:56:43.510772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636235+0000","last_change":"2023-11-24T06:20:57.459353+0000","last_active":"2023-11-24T06:20:58.636235+0000","last_peered":"2023-11-24T06:20:58.636235+0000","last_clean":"2023-11-24T06:20:58.636235+0000","last_became_active":"2023-11-24T06:20:57.459126+0000","last_became_peered":"2023-11-24T06:20:57.459126+0000","last_unstale":"2023-11-24T06:20:58.636235+0000","last_undegraded":"2023-11-24T06:20:58.636235+0000","last_fullsized":"2023-11-24T06:20:58.636235+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:07:24.523148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.439652+0000","last_change":"2023-11-24T06:20:57.449013+0000","last_active":"2023-11-24T06:20:58.439652+0000","last_peered":"2023-11-24T06:20:58.439652+0000","last_clean":"2023-11-24T06:20:58.439652+0000","last_became_active":"2023-11-24T06:20:57.448761+0000","last_became_peered":"2023-11-24T06:20:57.448761+0000","last_unstale":"2023-11-24T06:20:58.439652+0000","last_undegraded":"2023-11-24T06:20:58.439652+0000","last_fullsized":"2023-11-24T06:20:58.439652+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:15:44.765315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636398+0000","last_change":"2023-11-24T06:20:57.461781+0000","last_active":"2023-11-24T06:20:58.636398+0000","last_peered":"2023-11-24T06:20:58.636398+0000","last_clean":"2023-11-24T06:20:58.636398+0000","last_became_active":"2023-11-24T06:20:57.461496+0000","last_became_peered":"2023-11-24T06:20:57.461496+0000","last_unstale":"2023-11-24T06:20:58.636398+0000","last_undegraded":"2023-11-24T06:20:58.636398+0000","last_fullsized":"2023-11-24T06:20:58.636398+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:27:55.730669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636462+0000","last_change":"2023-11-24T06:20:57.462896+0000","last_active":"2023-11-24T06:20:58.636462+0000","last_peered":"2023-11-24T06:20:58.636462+0000","last_clean":"2023-11-24T06:20:58.636462+0000","last_became_active":"2023-11-24T06:20:57.462630+0000","last_became_peered":"2023-11-24T06:20:57.462630+0000","last_unstale":"2023-11-24T06:20:58.636462+0000","last_undegraded":"2023-11-24T06:20:58.636462+0000","last_fullsized":"2023-11-24T06:20:58.636462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:10.717250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.635706+0000","last_change":"2023-11-24T06:20:57.452852+0000","last_active":"2023-11-24T06:20:58.635706+0000","last_peered":"2023-11-24T06:20:58.635706+0000","last_clean":"2023-11-24T06:20:58.635706+0000","last_became_active":"2023-11-24T06:20:57.452173+0000","last_became_peered":"2023-11-24T06:20:57.452173+0000","last_unstale":"2023-11-24T06:20:58.635706+0000","last_undegraded":"2023-11-24T06:20:58.635706+0000","last_fullsized":"2023-11-24T06:20:58.635706+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:29.941324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637151+0000","last_change":"2023-11-24T06:20:57.455519+0000","last_active":"2023-11-24T06:20:58.637151+0000","last_peered":"2023-11-24T06:20:58.637151+0000","last_clean":"2023-11-24T06:20:58.637151+0000","last_became_active":"2023-11-24T06:20:57.453977+0000","last_became_peered":"2023-11-24T06:20:57.453977+0000","last_unstale":"2023-11-24T06:20:58.637151+0000","last_undegraded":"2023-11-24T06:20:58.637151+0000","last_fullsized":"2023-11-24T06:20:58.637151+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:40:16.265148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.439761+0000","last_change":"2023-11-24T06:20:57.459514+0000","last_active":"2023-11-24T06:20:58.439761+0000","last_peered":"2023-11-24T06:20:58.439761+0000","last_clean":"2023-11-24T06:20:58.439761+0000","last_became_active":"2023-11-24T06:20:57.459266+0000","last_became_peered":"2023-11-24T06:20:57.459266+0000","last_unstale":"2023-11-24T06:20:58.439761+0000","last_undegraded":"2023-11-24T06:20:58.439761+0000","last_fullsized":"2023-11-24T06:20:58.439761+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:39:15.047573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637844+0000","last_change":"2023-11-24T06:20:57.454647+0000","last_active":"2023-11-24T06:20:58.637844+0000","last_peered":"2023-11-24T06:20:58.637844+0000","last_clean":"2023-11-24T06:20:58.637844+0000","last_became_active":"2023-11-24T06:20:57.454420+0000","last_became_peered":"2023-11-24T06:20:57.454420+0000","last_unstale":"2023-11-24T06:20:58.637844+0000","last_undegraded":"2023-11-24T06:20:58.637844+0000","last_fullsized":"2023-11-24T06:20:58.637844+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:39:16.827953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637717+0000","last_change":"2023-11-24T06:20:57.449682+0000","last_active":"2023-11-24T06:20:58.637717+0000","last_peered":"2023-11-24T06:20:58.637717+0000","last_clean":"2023-11-24T06:20:58.637717+0000","last_became_active":"2023-11-24T06:20:57.449377+0000","last_became_peered":"2023-11-24T06:20:57.449377+0000","last_unstale":"2023-11-24T06:20:58.637717+0000","last_undegraded":"2023-11-24T06:20:58.637717+0000","last_fullsized":"2023-11-24T06:20:58.637717+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:12:58.275592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.440970+0000","last_change":"2023-11-24T06:20:57.460991+0000","last_active":"2023-11-24T06:20:58.440970+0000","last_peered":"2023-11-24T06:20:58.440970+0000","last_clean":"2023-11-24T06:20:58.440970+0000","last_became_active":"2023-11-24T06:20:57.460497+0000","last_became_peered":"2023-11-24T06:20:57.460497+0000","last_unstale":"2023-11-24T06:20:58.440970+0000","last_undegraded":"2023-11-24T06:20:58.440970+0000","last_fullsized":"2023-11-24T06:20:58.440970+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:34:21.521672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.440792+0000","last_change":"2023-11-24T06:20:57.452319+0000","last_active":"2023-11-24T06:20:58.440792+0000","last_peered":"2023-11-24T06:20:58.440792+0000","last_clean":"2023-11-24T06:20:58.440792+0000","last_became_active":"2023-11-24T06:20:57.452011+0000","last_became_peered":"2023-11-24T06:20:57.452011+0000","last_unstale":"2023-11-24T06:20:58.440792+0000","last_undegraded":"2023-11-24T06:20:58.440792+0000","last_fullsized":"2023-11-24T06:20:58.440792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:59:02.230504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.638096+0000","last_change":"2023-11-24T06:20:57.462295+0000","last_active":"2023-11-24T06:20:58.638096+0000","last_peered":"2023-11-24T06:20:58.638096+0000","last_clean":"2023-11-24T06:20:58.638096+0000","last_became_active":"2023-11-24T06:20:57.462037+0000","last_became_peered":"2023-11-24T06:20:57.462037+0000","last_unstale":"2023-11-24T06:20:58.638096+0000","last_undegraded":"2023-11-24T06:20:58.638096+0000","last_fullsized":"2023-11-24T06:20:58.638096+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:03:51.753199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637292+0000","last_change":"2023-11-24T06:20:57.459184+0000","last_active":"2023-11-24T06:20:58.637292+0000","last_peered":"2023-11-24T06:20:58.637292+0000","last_clean":"2023-11-24T06:20:58.637292+0000","last_became_active":"2023-11-24T06:20:57.458885+0000","last_became_peered":"2023-11-24T06:20:57.458885+0000","last_unstale":"2023-11-24T06:20:58.637292+0000","last_undegraded":"2023-11-24T06:20:58.637292+0000","last_fullsized":"2023-11-24T06:20:58.637292+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:16:55.703826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.437243+0000","last_change":"2023-11-24T06:20:57.457076+0000","last_active":"2023-11-24T06:20:58.437243+0000","last_peered":"2023-11-24T06:20:58.437243+0000","last_clean":"2023-11-24T06:20:58.437243+0000","last_became_active":"2023-11-24T06:20:57.456883+0000","last_became_peered":"2023-11-24T06:20:57.456883+0000","last_unstale":"2023-11-24T06:20:58.437243+0000","last_undegraded":"2023-11-24T06:20:58.437243+0000","last_fullsized":"2023-11-24T06:20:58.437243+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:53:58.765228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.638063+0000","last_change":"2023-11-24T06:20:57.450420+0000","last_active":"2023-11-24T06:20:58.638063+0000","last_peered":"2023-11-24T06:20:58.638063+0000","last_clean":"2023-11-24T06:20:58.638063+0000","last_became_active":"2023-11-24T06:20:57.450140+0000","last_became_peered":"2023-11-24T06:20:57.450140+0000","last_unstale":"2023-11-24T06:20:58.638063+0000","last_undegraded":"2023-11-24T06:20:58.638063+0000","last_fullsized":"2023-11-24T06:20:58.638063+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:59:26.373341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.436944+0000","last_change":"2023-11-24T06:20:57.458796+0000","last_active":"2023-11-24T06:20:58.436944+0000","last_peered":"2023-11-24T06:20:58.436944+0000","last_clean":"2023-11-24T06:20:58.436944+0000","last_became_active":"2023-11-24T06:20:57.458595+0000","last_became_peered":"2023-11-24T06:20:57.458595+0000","last_unstale":"2023-11-24T06:20:58.436944+0000","last_undegraded":"2023-11-24T06:20:58.436944+0000","last_fullsized":"2023-11-24T06:20:58.436944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:34:40.565750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636842+0000","last_change":"2023-11-24T06:20:57.465370+0000","last_active":"2023-11-24T06:20:58.636842+0000","last_peered":"2023-11-24T06:20:58.636842+0000","last_clean":"2023-11-24T06:20:58.636842+0000","last_became_active":"2023-11-24T06:20:57.465042+0000","last_became_peered":"2023-11-24T06:20:57.465042+0000","last_unstale":"2023-11-24T06:20:58.636842+0000","last_undegraded":"2023-11-24T06:20:58.636842+0000","last_fullsized":"2023-11-24T06:20:58.636842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:33:00.671048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636176+0000","last_change":"2023-11-24T06:20:57.459932+0000","last_active":"2023-11-24T06:20:58.636176+0000","last_peered":"2023-11-24T06:20:58.636176+0000","last_clean":"2023-11-24T06:20:58.636176+0000","last_became_active":"2023-11-24T06:20:57.458682+0000","last_became_peered":"2023-11-24T06:20:57.458682+0000","last_unstale":"2023-11-24T06:20:58.636176+0000","last_undegraded":"2023-11-24T06:20:58.636176+0000","last_fullsized":"2023-11-24T06:20:58.636176+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:56:43.510772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636617+0000","last_change":"2023-11-24T06:20:57.463485+0000","last_active":"2023-11-24T06:20:58.636617+0000","last_peered":"2023-11-24T06:20:58.636617+0000","last_clean":"2023-11-24T06:20:58.636617+0000","last_became_active":"2023-11-24T06:20:57.462945+0000","last_became_peered":"2023-11-24T06:20:57.462945+0000","last_unstale":"2023-11-24T06:20:58.636617+0000","last_undegraded":"2023-11-24T06:20:58.636617+0000","last_fullsized":"2023-11-24T06:20:58.636617+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:30:39.984756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441274+0000","last_change":"2023-11-24T06:20:57.456871+0000","last_active":"2023-11-24T06:20:58.441274+0000","last_peered":"2023-11-24T06:20:58.441274+0000","last_clean":"2023-11-24T06:20:58.441274+0000","last_became_active":"2023-11-24T06:20:57.456603+0000","last_became_peered":"2023-11-24T06:20:57.456603+0000","last_unstale":"2023-11-24T06:20:58.441274+0000","last_undegraded":"2023-11-24T06:20:58.441274+0000","last_fullsized":"2023-11-24T06:20:58.441274+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:14:18.177789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"63'4","reported_seq":16,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.457159+0000","last_change":"2023-11-24T06:20:56.450056+0000","last_active":"2023-11-24T06:20:57.457159+0000","last_peered":"2023-11-24T06:20:57.457159+0000","last_clean":"2023-11-24T06:20:57.457159+0000","last_became_active":"2023-11-24T06:20:56.449760+0000","last_became_peered":"2023-11-24T06:20:56.449760+0000","last_unstale":"2023-11-24T06:20:57.457159+0000","last_undegraded":"2023-11-24T06:20:57.457159+0000","last_fullsized":"2023-11-24T06:20:57.457159+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:31.618289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636540+0000","last_change":"2023-11-24T06:20:57.455059+0000","last_active":"2023-11-24T06:20:58.636540+0000","last_peered":"2023-11-24T06:20:58.636540+0000","last_clean":"2023-11-24T06:20:58.636540+0000","last_became_active":"2023-11-24T06:20:57.454778+0000","last_became_peered":"2023-11-24T06:20:57.454778+0000","last_unstale":"2023-11-24T06:20:58.636540+0000","last_undegraded":"2023-11-24T06:20:58.636540+0000","last_fullsized":"2023-11-24T06:20:58.636540+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:56:19.719773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.454118+0000","last_change":"2023-11-24T06:20:56.450515+0000","last_active":"2023-11-24T06:20:57.454118+0000","last_peered":"2023-11-24T06:20:57.454118+0000","last_clean":"2023-11-24T06:20:57.454118+0000","last_became_active":"2023-11-24T06:20:56.450219+0000","last_became_peered":"2023-11-24T06:20:56.450219+0000","last_unstale":"2023-11-24T06:20:57.454118+0000","last_undegraded":"2023-11-24T06:20:57.454118+0000","last_fullsized":"2023-11-24T06:20:57.454118+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:59:25.371715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637253+0000","last_change":"2023-11-24T06:20:57.466883+0000","last_active":"2023-11-24T06:20:58.637253+0000","last_peered":"2023-11-24T06:20:58.637253+0000","last_clean":"2023-11-24T06:20:58.637253+0000","last_became_active":"2023-11-24T06:20:57.466630+0000","last_became_peered":"2023-11-24T06:20:57.466630+0000","last_unstale":"2023-11-24T06:20:58.637253+0000","last_undegraded":"2023-11-24T06:20:58.637253+0000","last_fullsized":"2023-11-24T06:20:58.637253+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:45.946664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.467737+0000","last_change":"2023-11-24T06:20:56.457862+0000","last_active":"2023-11-24T06:20:57.467737+0000","last_peered":"2023-11-24T06:20:57.467737+0000","last_clean":"2023-11-24T06:20:57.467737+0000","last_became_active":"2023-11-24T06:20:56.457485+0000","last_became_peered":"2023-11-24T06:20:56.457485+0000","last_unstale":"2023-11-24T06:20:57.467737+0000","last_undegraded":"2023-11-24T06:20:57.467737+0000","last_fullsized":"2023-11-24T06:20:57.467737+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:44.945038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441260+0000","last_change":"2023-11-24T06:20:57.457167+0000","last_active":"2023-11-24T06:20:58.441260+0000","last_peered":"2023-11-24T06:20:58.441260+0000","last_clean":"2023-11-24T06:20:58.441260+0000","last_became_active":"2023-11-24T06:20:57.456858+0000","last_became_peered":"2023-11-24T06:20:57.456858+0000","last_unstale":"2023-11-24T06:20:58.441260+0000","last_undegraded":"2023-11-24T06:20:58.441260+0000","last_fullsized":"2023-11-24T06:20:58.441260+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:42:05.595322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.463297+0000","last_change":"2023-11-24T06:20:56.456592+0000","last_active":"2023-11-24T06:20:57.463297+0000","last_peered":"2023-11-24T06:20:57.463297+0000","last_clean":"2023-11-24T06:20:57.463297+0000","last_became_active":"2023-11-24T06:20:56.456416+0000","last_became_peered":"2023-11-24T06:20:56.456416+0000","last_unstale":"2023-11-24T06:20:57.463297+0000","last_undegraded":"2023-11-24T06:20:57.463297+0000","last_fullsized":"2023-11-24T06:20:57.463297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:09.995521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636905+0000","last_change":"2023-11-24T06:20:57.464112+0000","last_active":"2023-11-24T06:20:58.636905+0000","last_peered":"2023-11-24T06:20:58.636905+0000","last_clean":"2023-11-24T06:20:58.636905+0000","last_became_active":"2023-11-24T06:20:57.463835+0000","last_became_peered":"2023-11-24T06:20:57.463835+0000","last_unstale":"2023-11-24T06:20:58.636905+0000","last_undegraded":"2023-11-24T06:20:58.636905+0000","last_fullsized":"2023-11-24T06:20:58.636905+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:07.334941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.465589+0000","last_change":"2023-11-24T06:20:56.452696+0000","last_active":"2023-11-24T06:20:57.465589+0000","last_peered":"2023-11-24T06:20:57.465589+0000","last_clean":"2023-11-24T06:20:57.465589+0000","last_became_active":"2023-11-24T06:20:56.452226+0000","last_became_peered":"2023-11-24T06:20:56.452226+0000","last_unstale":"2023-11-24T06:20:57.465589+0000","last_undegraded":"2023-11-24T06:20:57.465589+0000","last_fullsized":"2023-11-24T06:20:57.465589+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:01.667702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.638218+0000","last_change":"2023-11-24T06:20:57.462855+0000","last_active":"2023-11-24T06:20:58.638218+0000","last_peered":"2023-11-24T06:20:58.638218+0000","last_clean":"2023-11-24T06:20:58.638218+0000","last_became_active":"2023-11-24T06:20:57.462363+0000","last_became_peered":"2023-11-24T06:20:57.462363+0000","last_unstale":"2023-11-24T06:20:58.638218+0000","last_undegraded":"2023-11-24T06:20:58.638218+0000","last_fullsized":"2023-11-24T06:20:58.638218+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:29.317739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.464341+0000","last_change":"2023-11-24T06:20:56.459294+0000","last_active":"2023-11-24T06:20:57.464341+0000","last_peered":"2023-11-24T06:20:57.464341+0000","last_clean":"2023-11-24T06:20:57.464341+0000","last_became_active":"2023-11-24T06:20:56.458979+0000","last_became_peered":"2023-11-24T06:20:56.458979+0000","last_unstale":"2023-11-24T06:20:57.464341+0000","last_undegraded":"2023-11-24T06:20:57.464341+0000","last_fullsized":"2023-11-24T06:20:57.464341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:09.069247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.635922+0000","last_change":"2023-11-24T06:20:57.458349+0000","last_active":"2023-11-24T06:20:58.635922+0000","last_peered":"2023-11-24T06:20:58.635922+0000","last_clean":"2023-11-24T06:20:58.635922+0000","last_became_active":"2023-11-24T06:20:57.458068+0000","last_became_peered":"2023-11-24T06:20:57.458068+0000","last_unstale":"2023-11-24T06:20:58.635922+0000","last_undegraded":"2023-11-24T06:20:58.635922+0000","last_fullsized":"2023-11-24T06:20:58.635922+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:41:06.209405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.441544+0000","last_change":"2023-11-24T06:20:56.451454+0000","last_active":"2023-11-24T06:20:57.441544+0000","last_peered":"2023-11-24T06:20:57.441544+0000","last_clean":"2023-11-24T06:20:57.441544+0000","last_became_active":"2023-11-24T06:20:56.451064+0000","last_became_peered":"2023-11-24T06:20:56.451064+0000","last_unstale":"2023-11-24T06:20:57.441544+0000","last_undegraded":"2023-11-24T06:20:57.441544+0000","last_fullsized":"2023-11-24T06:20:57.441544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:28.030140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.439794+0000","last_change":"2023-11-24T06:20:57.460350+0000","last_active":"2023-11-24T06:20:58.439794+0000","last_peered":"2023-11-24T06:20:58.439794+0000","last_clean":"2023-11-24T06:20:58.439794+0000","last_became_active":"2023-11-24T06:20:57.460079+0000","last_became_peered":"2023-11-24T06:20:57.460079+0000","last_unstale":"2023-11-24T06:20:58.439794+0000","last_undegraded":"2023-11-24T06:20:58.439794+0000","last_fullsized":"2023-11-24T06:20:58.439794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:34.947730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.436786+0000","last_change":"2023-11-24T06:20:56.449997+0000","last_active":"2023-11-24T06:20:57.436786+0000","last_peered":"2023-11-24T06:20:57.436786+0000","last_clean":"2023-11-24T06:20:57.436786+0000","last_became_active":"2023-11-24T06:20:56.449727+0000","last_became_peered":"2023-11-24T06:20:56.449727+0000","last_unstale":"2023-11-24T06:20:57.436786+0000","last_undegraded":"2023-11-24T06:20:57.436786+0000","last_fullsized":"2023-11-24T06:20:57.436786+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:20:07.394235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637440+0000","last_change":"2023-11-24T06:20:57.461428+0000","last_active":"2023-11-24T06:20:58.637440+0000","last_peered":"2023-11-24T06:20:58.637440+0000","last_clean":"2023-11-24T06:20:58.637440+0000","last_became_active":"2023-11-24T06:20:57.461129+0000","last_became_peered":"2023-11-24T06:20:57.461129+0000","last_unstale":"2023-11-24T06:20:58.637440+0000","last_undegraded":"2023-11-24T06:20:58.637440+0000","last_fullsized":"2023-11-24T06:20:58.637440+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:15.242814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.452535+0000","last_change":"2023-11-24T06:20:56.444835+0000","last_active":"2023-11-24T06:20:57.452535+0000","last_peered":"2023-11-24T06:20:57.452535+0000","last_clean":"2023-11-24T06:20:57.452535+0000","last_became_active":"2023-11-24T06:20:56.444547+0000","last_became_peered":"2023-11-24T06:20:56.444547+0000","last_unstale":"2023-11-24T06:20:57.452535+0000","last_undegraded":"2023-11-24T06:20:57.452535+0000","last_fullsized":"2023-11-24T06:20:57.452535+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:22:26.548969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441653+0000","last_change":"2023-11-24T06:20:57.461082+0000","last_active":"2023-11-24T06:20:58.441653+0000","last_peered":"2023-11-24T06:20:58.441653+0000","last_clean":"2023-11-24T06:20:58.441653+0000","last_became_active":"2023-11-24T06:20:57.460800+0000","last_became_peered":"2023-11-24T06:20:57.460800+0000","last_unstale":"2023-11-24T06:20:58.441653+0000","last_undegraded":"2023-11-24T06:20:58.441653+0000","last_fullsized":"2023-11-24T06:20:58.441653+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:39:52.225834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.449476+0000","last_change":"2023-11-24T06:20:56.455466+0000","last_active":"2023-11-24T06:20:57.449476+0000","last_peered":"2023-11-24T06:20:57.449476+0000","last_clean":"2023-11-24T06:20:57.449476+0000","last_became_active":"2023-11-24T06:20:56.455137+0000","last_became_peered":"2023-11-24T06:20:56.455137+0000","last_unstale":"2023-11-24T06:20:57.449476+0000","last_undegraded":"2023-11-24T06:20:57.449476+0000","last_fullsized":"2023-11-24T06:20:57.449476+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:28.030140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637395+0000","last_change":"2023-11-24T06:20:57.463808+0000","last_active":"2023-11-24T06:20:58.637395+0000","last_peered":"2023-11-24T06:20:58.637395+0000","last_clean":"2023-11-24T06:20:58.637395+0000","last_became_active":"2023-11-24T06:20:57.463539+0000","last_became_peered":"2023-11-24T06:20:57.463539+0000","last_unstale":"2023-11-24T06:20:58.637395+0000","last_undegraded":"2023-11-24T06:20:58.637395+0000","last_fullsized":"2023-11-24T06:20:58.637395+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:22:27.550594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"63'3","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.464839+0000","last_change":"2023-11-24T06:20:56.454452+0000","last_active":"2023-11-24T06:20:57.464839+0000","last_peered":"2023-11-24T06:20:57.464839+0000","last_clean":"2023-11-24T06:20:57.464839+0000","last_became_active":"2023-11-24T06:20:56.454143+0000","last_became_peered":"2023-11-24T06:20:56.454143+0000","last_unstale":"2023-11-24T06:20:57.464839+0000","last_undegraded":"2023-11-24T06:20:57.464839+0000","last_fullsized":"2023-11-24T06:20:57.464839+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:39:51.224209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636219+0000","last_change":"2023-11-24T06:20:57.445760+0000","last_active":"2023-11-24T06:20:58.636219+0000","last_peered":"2023-11-24T06:20:58.636219+0000","last_clean":"2023-11-24T06:20:58.636219+0000","last_became_active":"2023-11-24T06:20:57.445500+0000","last_became_peered":"2023-11-24T06:20:57.445500+0000","last_unstale":"2023-11-24T06:20:58.636219+0000","last_undegraded":"2023-11-24T06:20:58.636219+0000","last_fullsized":"2023-11-24T06:20:58.636219+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:21.283240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"63'6","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.472203+0000","last_change":"2023-11-24T06:20:56.450022+0000","last_active":"2023-11-24T06:20:57.472203+0000","last_peered":"2023-11-24T06:20:57.472203+0000","last_clean":"2023-11-24T06:20:57.472203+0000","last_became_active":"2023-11-24T06:20:56.449589+0000","last_became_peered":"2023-11-24T06:20:56.449589+0000","last_unstale":"2023-11-24T06:20:57.472203+0000","last_undegraded":"2023-11-24T06:20:57.472203+0000","last_fullsized":"2023-11-24T06:20:57.472203+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:49:20.354741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441035+0000","last_change":"2023-11-24T06:20:57.444542+0000","last_active":"2023-11-24T06:20:58.441035+0000","last_peered":"2023-11-24T06:20:58.441035+0000","last_clean":"2023-11-24T06:20:58.441035+0000","last_became_active":"2023-11-24T06:20:57.444265+0000","last_became_peered":"2023-11-24T06:20:57.444265+0000","last_unstale":"2023-11-24T06:20:58.441035+0000","last_undegraded":"2023-11-24T06:20:58.441035+0000","last_fullsized":"2023-11-24T06:20:58.441035+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:03.745129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.466022+0000","last_change":"2023-11-24T06:20:56.448978+0000","last_active":"2023-11-24T06:20:57.466022+0000","last_peered":"2023-11-24T06:20:57.466022+0000","last_clean":"2023-11-24T06:20:57.466022+0000","last_became_active":"2023-11-24T06:20:56.448657+0000","last_became_peered":"2023-11-24T06:20:56.448657+0000","last_unstale":"2023-11-24T06:20:57.466022+0000","last_undegraded":"2023-11-24T06:20:57.466022+0000","last_fullsized":"2023-11-24T06:20:57.466022+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:43:36.157888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636563+0000","last_change":"2023-11-24T06:20:57.458090+0000","last_active":"2023-11-24T06:20:58.636563+0000","last_peered":"2023-11-24T06:20:58.636563+0000","last_clean":"2023-11-24T06:20:58.636563+0000","last_became_active":"2023-11-24T06:20:57.457767+0000","last_became_peered":"2023-11-24T06:20:57.457767+0000","last_unstale":"2023-11-24T06:20:58.636563+0000","last_undegraded":"2023-11-24T06:20:58.636563+0000","last_fullsized":"2023-11-24T06:20:58.636563+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:43.414494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.442454+0000","last_change":"2023-11-24T06:20:56.455610+0000","last_active":"2023-11-24T06:20:57.442454+0000","last_peered":"2023-11-24T06:20:57.442454+0000","last_clean":"2023-11-24T06:20:57.442454+0000","last_became_active":"2023-11-24T06:20:56.455289+0000","last_became_peered":"2023-11-24T06:20:56.455289+0000","last_unstale":"2023-11-24T06:20:57.442454+0000","last_undegraded":"2023-11-24T06:20:57.442454+0000","last_fullsized":"2023-11-24T06:20:57.442454+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:14:08.909209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.439590+0000","last_change":"2023-11-24T06:20:57.461821+0000","last_active":"2023-11-24T06:20:58.439590+0000","last_peered":"2023-11-24T06:20:58.439590+0000","last_clean":"2023-11-24T06:20:58.439590+0000","last_became_active":"2023-11-24T06:20:57.461491+0000","last_became_peered":"2023-11-24T06:20:57.461491+0000","last_unstale":"2023-11-24T06:20:58.439590+0000","last_undegraded":"2023-11-24T06:20:58.439590+0000","last_fullsized":"2023-11-24T06:20:58.439590+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:07:29.317739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.447358+0000","last_change":"2023-11-24T06:20:56.455905+0000","last_active":"2023-11-24T06:20:57.447358+0000","last_peered":"2023-11-24T06:20:57.447358+0000","last_clean":"2023-11-24T06:20:57.447358+0000","last_became_active":"2023-11-24T06:20:56.455373+0000","last_became_peered":"2023-11-24T06:20:56.455373+0000","last_unstale":"2023-11-24T06:20:57.447358+0000","last_undegraded":"2023-11-24T06:20:57.447358+0000","last_fullsized":"2023-11-24T06:20:57.447358+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:15:43.763690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636088+0000","last_change":"2023-11-24T06:20:57.454489+0000","last_active":"2023-11-24T06:20:58.636088+0000","last_peered":"2023-11-24T06:20:58.636088+0000","last_clean":"2023-11-24T06:20:58.636088+0000","last_became_active":"2023-11-24T06:20:57.454235+0000","last_became_peered":"2023-11-24T06:20:57.454235+0000","last_unstale":"2023-11-24T06:20:58.636088+0000","last_undegraded":"2023-11-24T06:20:58.636088+0000","last_fullsized":"2023-11-24T06:20:58.636088+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:11:27.150377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"63'8","reported_seq":20,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.466931+0000","last_change":"2023-11-24T06:20:56.445784+0000","last_active":"2023-11-24T06:20:57.466931+0000","last_peered":"2023-11-24T06:20:57.466931+0000","last_clean":"2023-11-24T06:20:57.466931+0000","last_became_active":"2023-11-24T06:20:56.445468+0000","last_became_peered":"2023-11-24T06:20:56.445468+0000","last_unstale":"2023-11-24T06:20:57.466931+0000","last_undegraded":"2023-11-24T06:20:57.466931+0000","last_fullsized":"2023-11-24T06:20:57.466931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:40:15.263523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.638087+0000","last_change":"2023-11-24T06:20:57.453548+0000","last_active":"2023-11-24T06:20:58.638087+0000","last_peered":"2023-11-24T06:20:58.638087+0000","last_clean":"2023-11-24T06:20:58.638087+0000","last_became_active":"2023-11-24T06:20:57.453345+0000","last_became_peered":"2023-11-24T06:20:57.453345+0000","last_unstale":"2023-11-24T06:20:58.638087+0000","last_undegraded":"2023-11-24T06:20:58.638087+0000","last_fullsized":"2023-11-24T06:20:58.638087+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:30.473445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.462213+0000","last_change":"2023-11-24T06:20:56.458379+0000","last_active":"2023-11-24T06:20:57.462213+0000","last_peered":"2023-11-24T06:20:57.462213+0000","last_clean":"2023-11-24T06:20:57.462213+0000","last_became_active":"2023-11-24T06:20:56.457956+0000","last_became_peered":"2023-11-24T06:20:56.457956+0000","last_unstale":"2023-11-24T06:20:57.462213+0000","last_undegraded":"2023-11-24T06:20:57.462213+0000","last_fullsized":"2023-11-24T06:20:57.462213+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:43:36.157888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441784+0000","last_change":"2023-11-24T06:20:57.460076+0000","last_active":"2023-11-24T06:20:58.441784+0000","last_peered":"2023-11-24T06:20:58.441784+0000","last_clean":"2023-11-24T06:20:58.441784+0000","last_became_active":"2023-11-24T06:20:57.456480+0000","last_became_peered":"2023-11-24T06:20:57.456480+0000","last_unstale":"2023-11-24T06:20:58.441784+0000","last_undegraded":"2023-11-24T06:20:58.441784+0000","last_fullsized":"2023-11-24T06:20:58.441784+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:13:57.997147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.467739+0000","last_change":"2023-11-24T06:20:56.459307+0000","last_active":"2023-11-24T06:20:57.467739+0000","last_peered":"2023-11-24T06:20:57.467739+0000","last_clean":"2023-11-24T06:20:57.467739+0000","last_became_active":"2023-11-24T06:20:56.459077+0000","last_became_peered":"2023-11-24T06:20:56.459077+0000","last_unstale":"2023-11-24T06:20:57.467739+0000","last_undegraded":"2023-11-24T06:20:57.467739+0000","last_fullsized":"2023-11-24T06:20:57.467739+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:17:50.968467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.635770+0000","last_change":"2023-11-24T06:20:57.449080+0000","last_active":"2023-11-24T06:20:58.635770+0000","last_peered":"2023-11-24T06:20:58.635770+0000","last_clean":"2023-11-24T06:20:58.635770+0000","last_became_active":"2023-11-24T06:20:57.448339+0000","last_became_peered":"2023-11-24T06:20:57.448339+0000","last_unstale":"2023-11-24T06:20:58.635770+0000","last_undegraded":"2023-11-24T06:20:58.635770+0000","last_fullsized":"2023-11-24T06:20:58.635770+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:45.946664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"63'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.467303+0000","last_change":"2023-11-24T06:20:56.457538+0000","last_active":"2023-11-24T06:20:57.467303+0000","last_peered":"2023-11-24T06:20:57.467303+0000","last_clean":"2023-11-24T06:20:57.467303+0000","last_became_active":"2023-11-24T06:20:56.457247+0000","last_became_peered":"2023-11-24T06:20:56.457247+0000","last_unstale":"2023-11-24T06:20:57.467303+0000","last_undegraded":"2023-11-24T06:20:57.467303+0000","last_fullsized":"2023-11-24T06:20:57.467303+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:49:03.890519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636309+0000","last_change":"2023-11-24T06:20:57.453530+0000","last_active":"2023-11-24T06:20:58.636309+0000","last_peered":"2023-11-24T06:20:58.636309+0000","last_clean":"2023-11-24T06:20:58.636309+0000","last_became_active":"2023-11-24T06:20:57.453224+0000","last_became_peered":"2023-11-24T06:20:57.453224+0000","last_unstale":"2023-11-24T06:20:58.636309+0000","last_undegraded":"2023-11-24T06:20:58.636309+0000","last_fullsized":"2023-11-24T06:20:58.636309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:14:09.910834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.437725+0000","last_change":"2023-11-24T06:20:56.453032+0000","last_active":"2023-11-24T06:20:57.437725+0000","last_peered":"2023-11-24T06:20:57.437725+0000","last_clean":"2023-11-24T06:20:57.437725+0000","last_became_active":"2023-11-24T06:20:56.452748+0000","last_became_peered":"2023-11-24T06:20:56.452748+0000","last_unstale":"2023-11-24T06:20:57.437725+0000","last_undegraded":"2023-11-24T06:20:57.437725+0000","last_fullsized":"2023-11-24T06:20:57.437725+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:14.241189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.437089+0000","last_change":"2023-11-24T06:20:57.453994+0000","last_active":"2023-11-24T06:20:58.437089+0000","last_peered":"2023-11-24T06:20:58.437089+0000","last_clean":"2023-11-24T06:20:58.437089+0000","last_became_active":"2023-11-24T06:20:57.453701+0000","last_became_peered":"2023-11-24T06:20:57.453701+0000","last_unstale":"2023-11-24T06:20:58.437089+0000","last_undegraded":"2023-11-24T06:20:58.437089+0000","last_fullsized":"2023-11-24T06:20:58.437089+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:32:16.619340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.446571+0000","last_change":"2023-11-24T06:20:56.459215+0000","last_active":"2023-11-24T06:20:57.446571+0000","last_peered":"2023-11-24T06:20:57.446571+0000","last_clean":"2023-11-24T06:20:57.446571+0000","last_became_active":"2023-11-24T06:20:56.458937+0000","last_became_peered":"2023-11-24T06:20:56.458937+0000","last_unstale":"2023-11-24T06:20:57.446571+0000","last_undegraded":"2023-11-24T06:20:57.446571+0000","last_fullsized":"2023-11-24T06:20:57.446571+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:57:07.713106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.436944+0000","last_change":"2023-11-24T06:20:57.454052+0000","last_active":"2023-11-24T06:20:58.436944+0000","last_peered":"2023-11-24T06:20:58.436944+0000","last_clean":"2023-11-24T06:20:58.436944+0000","last_became_active":"2023-11-24T06:20:57.453833+0000","last_became_peered":"2023-11-24T06:20:57.453833+0000","last_unstale":"2023-11-24T06:20:58.436944+0000","last_undegraded":"2023-11-24T06:20:58.436944+0000","last_fullsized":"2023-11-24T06:20:58.436944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:07:08.435642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.447500+0000","last_change":"2023-11-24T06:20:56.455809+0000","last_active":"2023-11-24T06:20:57.447500+0000","last_peered":"2023-11-24T06:20:57.447500+0000","last_clean":"2023-11-24T06:20:57.447500+0000","last_became_active":"2023-11-24T06:20:56.455461+0000","last_became_peered":"2023-11-24T06:20:56.455461+0000","last_unstale":"2023-11-24T06:20:57.447500+0000","last_undegraded":"2023-11-24T06:20:57.447500+0000","last_fullsized":"2023-11-24T06:20:57.447500+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:39:14.045947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.437364+0000","last_change":"2023-11-24T06:20:57.454792+0000","last_active":"2023-11-24T06:20:58.437364+0000","last_peered":"2023-11-24T06:20:58.437364+0000","last_clean":"2023-11-24T06:20:58.437364+0000","last_became_active":"2023-11-24T06:20:57.454424+0000","last_became_peered":"2023-11-24T06:20:57.454424+0000","last_unstale":"2023-11-24T06:20:58.437364+0000","last_undegraded":"2023-11-24T06:20:58.437364+0000","last_fullsized":"2023-11-24T06:20:58.437364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:18:33.947107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.443196+0000","last_change":"2023-11-24T06:20:56.450795+0000","last_active":"2023-11-24T06:20:57.443196+0000","last_peered":"2023-11-24T06:20:57.443196+0000","last_clean":"2023-11-24T06:20:57.443196+0000","last_became_active":"2023-11-24T06:20:56.450111+0000","last_became_peered":"2023-11-24T06:20:56.450111+0000","last_unstale":"2023-11-24T06:20:57.443196+0000","last_undegraded":"2023-11-24T06:20:57.443196+0000","last_fullsized":"2023-11-24T06:20:57.443196+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:27:54.729044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.439837+0000","last_change":"2023-11-24T06:20:57.453028+0000","last_active":"2023-11-24T06:20:58.439837+0000","last_peered":"2023-11-24T06:20:58.439837+0000","last_clean":"2023-11-24T06:20:58.439837+0000","last_became_active":"2023-11-24T06:20:57.452711+0000","last_became_peered":"2023-11-24T06:20:57.452711+0000","last_unstale":"2023-11-24T06:20:58.439837+0000","last_undegraded":"2023-11-24T06:20:58.439837+0000","last_fullsized":"2023-11-24T06:20:58.439837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:38.571288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.462085+0000","last_change":"2023-11-24T06:20:56.458175+0000","last_active":"2023-11-24T06:20:57.462085+0000","last_peered":"2023-11-24T06:20:57.462085+0000","last_clean":"2023-11-24T06:20:57.462085+0000","last_became_active":"2023-11-24T06:20:56.457570+0000","last_became_peered":"2023-11-24T06:20:56.457570+0000","last_unstale":"2023-11-24T06:20:57.462085+0000","last_undegraded":"2023-11-24T06:20:57.462085+0000","last_fullsized":"2023-11-24T06:20:57.462085+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:30:52.218412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636320+0000","last_change":"2023-11-24T06:20:57.458413+0000","last_active":"2023-11-24T06:20:58.636320+0000","last_peered":"2023-11-24T06:20:58.636320+0000","last_clean":"2023-11-24T06:20:58.636320+0000","last_became_active":"2023-11-24T06:20:57.458136+0000","last_became_peered":"2023-11-24T06:20:57.458136+0000","last_unstale":"2023-11-24T06:20:58.636320+0000","last_undegraded":"2023-11-24T06:20:58.636320+0000","last_fullsized":"2023-11-24T06:20:58.636320+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:38.571288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.446753+0000","last_change":"2023-11-24T06:20:56.454563+0000","last_active":"2023-11-24T06:20:57.446753+0000","last_peered":"2023-11-24T06:20:57.446753+0000","last_clean":"2023-11-24T06:20:57.446753+0000","last_became_active":"2023-11-24T06:20:56.454014+0000","last_became_peered":"2023-11-24T06:20:56.454014+0000","last_unstale":"2023-11-24T06:20:57.446753+0000","last_undegraded":"2023-11-24T06:20:57.446753+0000","last_fullsized":"2023-11-24T06:20:57.446753+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:03:50.751574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441173+0000","last_change":"2023-11-24T06:20:57.452725+0000","last_active":"2023-11-24T06:20:58.441173+0000","last_peered":"2023-11-24T06:20:58.441173+0000","last_clean":"2023-11-24T06:20:58.441173+0000","last_became_active":"2023-11-24T06:20:57.452449+0000","last_became_peered":"2023-11-24T06:20:57.452449+0000","last_unstale":"2023-11-24T06:20:58.441173+0000","last_undegraded":"2023-11-24T06:20:58.441173+0000","last_fullsized":"2023-11-24T06:20:58.441173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:41:20.826962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.437732+0000","last_change":"2023-11-24T06:20:56.455444+0000","last_active":"2023-11-24T06:20:57.437732+0000","last_peered":"2023-11-24T06:20:57.437732+0000","last_clean":"2023-11-24T06:20:57.437732+0000","last_became_active":"2023-11-24T06:20:56.455167+0000","last_became_peered":"2023-11-24T06:20:56.455167+0000","last_unstale":"2023-11-24T06:20:57.437732+0000","last_undegraded":"2023-11-24T06:20:57.437732+0000","last_fullsized":"2023-11-24T06:20:57.437732+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:36:20.310219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.441365+0000","last_change":"2023-11-24T06:20:57.462505+0000","last_active":"2023-11-24T06:20:58.441365+0000","last_peered":"2023-11-24T06:20:58.441365+0000","last_clean":"2023-11-24T06:20:58.441365+0000","last_became_active":"2023-11-24T06:20:57.460409+0000","last_became_peered":"2023-11-24T06:20:57.460409+0000","last_unstale":"2023-11-24T06:20:58.441365+0000","last_undegraded":"2023-11-24T06:20:58.441365+0000","last_fullsized":"2023-11-24T06:20:58.441365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:02.669327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.446562+0000","last_change":"2023-11-24T06:20:56.453405+0000","last_active":"2023-11-24T06:20:57.446562+0000","last_peered":"2023-11-24T06:20:57.446562+0000","last_clean":"2023-11-24T06:20:57.446562+0000","last_became_active":"2023-11-24T06:20:56.453077+0000","last_became_peered":"2023-11-24T06:20:56.453077+0000","last_unstale":"2023-11-24T06:20:57.446562+0000","last_undegraded":"2023-11-24T06:20:57.446562+0000","last_fullsized":"2023-11-24T06:20:57.446562+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:03:50.751574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636258+0000","last_change":"2023-11-24T06:20:57.463084+0000","last_active":"2023-11-24T06:20:58.636258+0000","last_peered":"2023-11-24T06:20:58.636258+0000","last_clean":"2023-11-24T06:20:58.636258+0000","last_became_active":"2023-11-24T06:20:57.462497+0000","last_became_peered":"2023-11-24T06:20:57.462497+0000","last_unstale":"2023-11-24T06:20:58.636258+0000","last_undegraded":"2023-11-24T06:20:58.636258+0000","last_fullsized":"2023-11-24T06:20:58.636258+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:28:10.070872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.437058+0000","last_change":"2023-11-24T06:20:56.455600+0000","last_active":"2023-11-24T06:20:57.437058+0000","last_peered":"2023-11-24T06:20:57.437058+0000","last_clean":"2023-11-24T06:20:57.437058+0000","last_became_active":"2023-11-24T06:20:56.455299+0000","last_became_peered":"2023-11-24T06:20:56.455299+0000","last_unstale":"2023-11-24T06:20:57.437058+0000","last_undegraded":"2023-11-24T06:20:57.437058+0000","last_fullsized":"2023-11-24T06:20:57.437058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:16:13.458374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.637096+0000","last_change":"2023-11-24T06:20:57.466653+0000","last_active":"2023-11-24T06:20:58.637096+0000","last_peered":"2023-11-24T06:20:58.637096+0000","last_clean":"2023-11-24T06:20:58.637096+0000","last_became_active":"2023-11-24T06:20:57.466409+0000","last_became_peered":"2023-11-24T06:20:57.466409+0000","last_unstale":"2023-11-24T06:20:58.637096+0000","last_undegraded":"2023-11-24T06:20:58.637096+0000","last_fullsized":"2023-11-24T06:20:58.637096+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:52:21.060199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.441019+0000","last_change":"2023-11-24T06:20:56.453053+0000","last_active":"2023-11-24T06:20:57.441019+0000","last_peered":"2023-11-24T06:20:57.441019+0000","last_clean":"2023-11-24T06:20:57.441019+0000","last_became_active":"2023-11-24T06:20:56.452778+0000","last_became_peered":"2023-11-24T06:20:56.452778+0000","last_unstale":"2023-11-24T06:20:57.441019+0000","last_undegraded":"2023-11-24T06:20:57.441019+0000","last_fullsized":"2023-11-24T06:20:57.441019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:44:24.187407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.635577+0000","last_change":"2023-11-24T06:20:57.461722+0000","last_active":"2023-11-24T06:20:58.635577+0000","last_peered":"2023-11-24T06:20:58.635577+0000","last_clean":"2023-11-24T06:20:58.635577+0000","last_became_active":"2023-11-24T06:20:57.461437+0000","last_became_peered":"2023-11-24T06:20:57.461437+0000","last_unstale":"2023-11-24T06:20:58.635577+0000","last_undegraded":"2023-11-24T06:20:58.635577+0000","last_fullsized":"2023-11-24T06:20:58.635577+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:52:21.060199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.450025+0000","last_change":"2023-11-24T06:20:56.445884+0000","last_active":"2023-11-24T06:20:57.450025+0000","last_peered":"2023-11-24T06:20:57.450025+0000","last_clean":"2023-11-24T06:20:57.450025+0000","last_became_active":"2023-11-24T06:20:56.445600+0000","last_became_peered":"2023-11-24T06:20:56.445600+0000","last_unstale":"2023-11-24T06:20:57.450025+0000","last_undegraded":"2023-11-24T06:20:57.450025+0000","last_fullsized":"2023-11-24T06:20:57.450025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:28.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.636732+0000","last_change":"2023-11-24T06:20:57.462526+0000","last_active":"2023-11-24T06:20:58.636732+0000","last_peered":"2023-11-24T06:20:58.636732+0000","last_clean":"2023-11-24T06:20:58.636732+0000","last_became_active":"2023-11-24T06:20:57.462247+0000","last_became_peered":"2023-11-24T06:20:57.462247+0000","last_unstale":"2023-11-24T06:20:58.636732+0000","last_undegraded":"2023-11-24T06:20:58.636732+0000","last_fullsized":"2023-11-24T06:20:58.636732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:49:21.356366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.445651+0000","last_change":"2023-11-24T06:20:56.456769+0000","last_active":"2023-11-24T06:20:57.445651+0000","last_peered":"2023-11-24T06:20:57.445651+0000","last_clean":"2023-11-24T06:20:57.445651+0000","last_became_active":"2023-11-24T06:20:56.456558+0000","last_became_peered":"2023-11-24T06:20:56.456558+0000","last_unstale":"2023-11-24T06:20:57.445651+0000","last_undegraded":"2023-11-24T06:20:57.445651+0000","last_fullsized":"2023-11-24T06:20:57.445651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:24:18.947469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.437127+0000","last_change":"2023-11-24T06:20:57.465287+0000","last_active":"2023-11-24T06:20:58.437127+0000","last_peered":"2023-11-24T06:20:58.437127+0000","last_clean":"2023-11-24T06:20:58.437127+0000","last_became_active":"2023-11-24T06:20:57.464988+0000","last_became_peered":"2023-11-24T06:20:57.464988+0000","last_unstale":"2023-11-24T06:20:58.437127+0000","last_undegraded":"2023-11-24T06:20:58.437127+0000","last_fullsized":"2023-11-24T06:20:58.437127+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:56.425380+0000","last_clean_scrub_stamp":"2023-11-24T06:20:56.425380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:10:27.651163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"63'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.453598+0000","last_change":"2023-11-24T06:20:56.456842+0000","last_active":"2023-11-24T06:20:57.453598+0000","last_peered":"2023-11-24T06:20:57.453598+0000","last_clean":"2023-11-24T06:20:57.453598+0000","last_became_active":"2023-11-24T06:20:56.456506+0000","last_became_peered":"2023-11-24T06:20:56.456506+0000","last_unstale":"2023-11-24T06:20:57.453598+0000","last_undegraded":"2023-11-24T06:20:57.453598+0000","last_fullsized":"2023-11-24T06:20:57.453598+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:20:55.423754+0000","last_clean_scrub_stamp":"2023-11-24T06:20:55.423754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:28:58.368153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":209,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-11-24T06:20:57.437410+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:57.437410+0000","last_peered":"2023-11-24T06:20:57.437410+0000","last_clean":"2023-11-24T06:20:57.437410+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:57.437410+0000","last_undegraded":"2023-11-24T06:20:57.437410+0000","last_fullsized":"2023-11-24T06:20:57.437410+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28376,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343464,"statfs":{"total":96636764160,"available":96607707136,"internally_reserved":0,"allocated":1380352,"data_stored":924602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.63800000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.56599999999999995}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.622}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55400000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":333710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.497},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.53600000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.46700000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.60299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":335846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.35799999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.311}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.371},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.28599999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.622},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.58599999999999997}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.56100000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.67100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343472,"statfs":{"total":96636764160,"available":96607715328,"internally_reserved":0,"allocated":1372160,"data_stored":923986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.29499999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.254}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32400000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.504},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.73699999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.71499999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":335230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.24099999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.307},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.29299999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.32700000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.58599999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.48299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":333730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.37}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.32100000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.85999999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.95799999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.80700000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.752},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.92100000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574882,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":333688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58899999999999997}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.64700000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.59999999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.46400000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.496}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.48099999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":333708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.72699999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.69199999999999995}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.73799999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.71599999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.754},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:06.486 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T06:21:06.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T06:21:06.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.647+0000 7f18451d1700 1 Processor -- start 2023-11-24T06:21:06.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.647+0000 7f18451d1700 1 -- start start 2023-11-24T06:21:06.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.648+0000 7f18451d1700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f18400b4370 0x7f184015b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.648+0000 7f18451d1700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18400ae5e0 con 0x7f18401587f0 2023-11-24T06:21:06.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.648+0000 7f18451d1700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184015be30 con 0x7f184015b370 2023-11-24T06:21:06.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.648+0000 7f18451d1700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184015c110 con 0x7f18400b4370 2023-11-24T06:21:06.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.648+0000 7f183f59e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f18400b4370 0x7f184015b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.648+0000 7f183f59e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f18400b4370 0x7f184015b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39072/0 (socket says 172.21.15.89:39072) 2023-11-24T06:21:06.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.648+0000 7f183f59e700 1 -- 172.21.15.89:0/4214329857 learned_addr learned my addr 172.21.15.89:0/4214329857 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:06.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.648+0000 7f183ed9d700 1 --1- 172.21.15.89:0/4214329857 >> v1:172.21.15.89:6789/0 conn(0x7f18401587f0 0x7f184006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42572/0 (socket says 172.21.15.89:42572) 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.649+0000 7f183f59e700 1 -- 172.21.15.89:0/4214329857 >> v1:172.21.15.123:6789/0 conn(0x7f184015b370 legacy=0x7f18400b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.649+0000 7f183f59e700 1 -- 172.21.15.89:0/4214329857 >> v1:172.21.15.89:6789/0 conn(0x7f18401587f0 legacy=0x7f184006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.649+0000 7f183f59e700 1 -- 172.21.15.89:0/4214329857 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18400ae860 con 0x7f18400b4370 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.649+0000 7f183dd9b700 1 -- 172.21.15.89:0/4214329857 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941298955 0 0) 0x7f18400ae5e0 con 0x7f18401587f0 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.649+0000 7f183f59e700 1 --2- 172.21.15.89:0/4214329857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184015b800 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1838009e80 tx=0x7f183800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc30899edc7786e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.649+0000 7f183dd9b700 1 -- 172.21.15.89:0/4214329857 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183800e070 con 0x7f18400b4370 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.649+0000 7f183dd9b700 1 -- 172.21.15.89:0/4214329857 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1838003a20 con 0x7f18400b4370 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.649+0000 7f183dd9b700 1 -- 172.21.15.89:0/4214329857 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838003dd0 con 0x7f18400b4370 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.650+0000 7f18451d1700 1 -- 172.21.15.89:0/4214329857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 msgr2=0x7f184015b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.650+0000 7f18451d1700 1 --2- 172.21.15.89:0/4214329857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184015b800 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f1838009e80 tx=0x7f183800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.650+0000 7f18451d1700 1 -- 172.21.15.89:0/4214329857 shutdown_connections 2023-11-24T06:21:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.650+0000 7f18451d1700 1 --2- 172.21.15.89:0/4214329857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184015b800 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.650+0000 7f18451d1700 1 -- 172.21.15.89:0/4214329857 >> 172.21.15.89:0/4214329857 conn(0x7f18400d0bf0 msgr2=0x7f18400d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.650+0000 7f18451d1700 1 -- 172.21.15.89:0/4214329857 shutdown_connections 2023-11-24T06:21:06.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.650+0000 7f18451d1700 1 -- 172.21.15.89:0/4214329857 wait complete. 2023-11-24T06:21:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.651+0000 7f18451d1700 1 Processor -- start 2023-11-24T06:21:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.651+0000 7f18451d1700 1 -- start start 2023-11-24T06:21:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f18451d1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184014b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f18451d1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18401587f0 0x7f18401f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f18451d1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f184015b370 0x7f18401f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f18451d1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18400b1a90 con 0x7f18400b4370 2023-11-24T06:21:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f18451d1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184014b580 con 0x7f18401587f0 2023-11-24T06:21:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f18451d1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1840148610 con 0x7f184015b370 2023-11-24T06:21:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184014b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184014b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39074/0 (socket says 172.21.15.89:39074) 2023-11-24T06:21:06.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 -- 172.21.15.89:0/360385007 learned_addr learned my addr 172.21.15.89:0/360385007 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183e59c700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18401587f0 0x7f18401f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183f59e700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f184015b370 0x7f18401f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 -- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f184015b370 msgr2=0x7f18401f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f184015b370 0x7f18401f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 -- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18401587f0 msgr2=0x7f18401f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18401587f0 0x7f18401f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 -- 172.21.15.89:0/360385007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1838009b30 con 0x7f18400b4370 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183e59c700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18401587f0 0x7f18401f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.652+0000 7f183ed9d700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184014b040 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f183400e7a0 tx=0x7f183400eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.653+0000 7f18277fe700 1 -- 172.21.15.89:0/360385007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1834006e90 con 0x7f18400b4370 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.653+0000 7f18277fe700 1 -- 172.21.15.89:0/360385007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18340090d0 con 0x7f18400b4370 2023-11-24T06:21:06.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.653+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1840145910 con 0x7f18400b4370 2023-11-24T06:21:06.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.653+0000 7f18277fe700 1 -- 172.21.15.89:0/360385007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18340094a0 con 0x7f18400b4370 2023-11-24T06:21:06.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.653+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18401452c0 con 0x7f18400b4370 2023-11-24T06:21:06.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.654+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1840145120 con 0x7f18400b4370 2023-11-24T06:21:06.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.654+0000 7f18277fe700 1 -- 172.21.15.89:0/360385007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f1834009600 con 0x7f18400b4370 2023-11-24T06:21:06.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.655+0000 7f18277fe700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f182c06af60 0x7f182c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:06.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.655+0000 7f183e59c700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f182c06af60 0x7f182c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:06.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.655+0000 7f18277fe700 1 -- 172.21.15.89:0/360385007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f183408ad40 con 0x7f18400b4370 2023-11-24T06:21:06.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.655+0000 7f183e59c700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f182c06af60 0x7f182c06d420 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1830006fd0 tx=0x7f1830008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:06.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.659+0000 7f18277fe700 1 -- 172.21.15.89:0/360385007 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1840145120 con 0x7f18400b4370 2023-11-24T06:21:06.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.930+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f1840145120 con 0x7f18400b4370 2023-11-24T06:21:06.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.932+0000 7f18277fe700 1 -- 172.21.15.89:0/360385007 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f1840145120 con 0x7f18400b4370 2023-11-24T06:21:06.934 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2023-11-24T06:21:06.934 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAOQGBlwIfyKhAAMZcwOQGAm+XcH+mUKcTiUg== 2023-11-24T06:21:06.935 INFO:teuthology.orchestra.run.smithi089.stdout: caps mds = "allow" 2023-11-24T06:21:06.935 INFO:teuthology.orchestra.run.smithi089.stdout: caps mon = "allow r" 2023-11-24T06:21:06.935 INFO:teuthology.orchestra.run.smithi089.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.934+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f182c06af60 msgr2=0x7f182c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.934+0000 7f18451d1700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f182c06af60 0x7f182c06d420 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1830006fd0 tx=0x7f1830008040 comp rx=0 tx=0).stop 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.934+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 msgr2=0x7f184014b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.934+0000 7f18451d1700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184014b040 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f183400e7a0 tx=0x7f183400eb60 comp rx=0 tx=0).stop 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.935+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 shutdown_connections 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.935+0000 7f18451d1700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f184015b370 0x7f18401f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.935+0000 7f18451d1700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18400b4370 0x7f184014b040 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.935+0000 7f18451d1700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f182c06af60 0x7f182c06d420 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.935+0000 7f18451d1700 1 --2- 172.21.15.89:0/360385007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18401587f0 0x7f18401f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:06.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.935+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 >> 172.21.15.89:0/360385007 conn(0x7f18400d0bf0 msgr2=0x7f18400af060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:06.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.935+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 shutdown_connections 2023-11-24T06:21:06.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:06.935+0000 7f18451d1700 1 -- 172.21.15.89:0/360385007 wait complete. 2023-11-24T06:21:06.938 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2023-11-24T06:21:06.949 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:21:07.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.067+0000 7fb29b767700 1 Processor -- start 2023-11-24T06:21:07.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.068+0000 7fb29b767700 1 -- start start 2023-11-24T06:21:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.068+0000 7fb29b767700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb29406c710 0x7fb2940b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.068+0000 7fb29b767700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2940ae430 con 0x7fb29415e080 2023-11-24T06:21:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.068+0000 7fb29b767700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2940ae710 con 0x7fb2940af4d0 2023-11-24T06:21:07.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.068+0000 7fb29b767700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2940af0b0 con 0x7fb29406c710 2023-11-24T06:21:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb298d02700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb29406c710 0x7fb2940b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb299503700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb29415e080 0x7fb29406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42586/0 (socket says 172.21.15.89:42586) 2023-11-24T06:21:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb299503700 1 -- 172.21.15.89:0/2204336055 learned_addr learned my addr 172.21.15.89:0/2204336055 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb298d02700 1 -- 172.21.15.89:0/2204336055 >> v1:172.21.15.123:6789/0 conn(0x7fb2940af4d0 legacy=0x7fb29406e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb298d02700 1 -- 172.21.15.89:0/2204336055 >> v1:172.21.15.89:6789/0 conn(0x7fb29415e080 legacy=0x7fb29406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb28bfff700 1 -- 172.21.15.89:0/2204336055 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284873615 0 0) 0x7fb2940ae430 con 0x7fb29415e080 2023-11-24T06:21:07.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb298d02700 1 -- 172.21.15.89:0/2204336055 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2940ae9f0 con 0x7fb29406c710 2023-11-24T06:21:07.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb298d02700 1 --2- 172.21.15.89:0/2204336055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb2940b5050 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb290009840 tx=0x7fb290009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5841f69145990c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb28bfff700 1 -- 172.21.15.89:0/2204336055 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290004230 con 0x7fb29406c710 2023-11-24T06:21:07.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb28bfff700 1 -- 172.21.15.89:0/2204336055 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb290004390 con 0x7fb29406c710 2023-11-24T06:21:07.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.069+0000 7fb28bfff700 1 -- 172.21.15.89:0/2204336055 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900036a0 con 0x7fb29406c710 2023-11-24T06:21:07.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.070+0000 7fb29b767700 1 -- 172.21.15.89:0/2204336055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 msgr2=0x7fb2940b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.070+0000 7fb29b767700 1 --2- 172.21.15.89:0/2204336055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb2940b5050 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb290009840 tx=0x7fb290009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:07.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.070+0000 7fb29b767700 1 -- 172.21.15.89:0/2204336055 shutdown_connections 2023-11-24T06:21:07.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.070+0000 7fb29b767700 1 --2- 172.21.15.89:0/2204336055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb2940b5050 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.070+0000 7fb29b767700 1 -- 172.21.15.89:0/2204336055 >> 172.21.15.89:0/2204336055 conn(0x7fb294071b90 msgr2=0x7fb294071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.070+0000 7fb29b767700 1 -- 172.21.15.89:0/2204336055 shutdown_connections 2023-11-24T06:21:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.070+0000 7fb29b767700 1 -- 172.21.15.89:0/2204336055 wait complete. 2023-11-24T06:21:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.071+0000 7fb29b767700 1 Processor -- start 2023-11-24T06:21:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.071+0000 7fb29b767700 1 -- start start 2023-11-24T06:21:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb29b767700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb294160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb29b767700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2940af4d0 0x7fb294153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb29b767700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29415e080 0x7fb2941f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb29b767700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294156380 con 0x7fb29406c710 2023-11-24T06:21:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb29b767700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294161460 con 0x7fb2940af4d0 2023-11-24T06:21:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb29b767700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb294156570 con 0x7fb29415e080 2023-11-24T06:21:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb294160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb294160d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39088/0 (socket says 172.21.15.89:39088) 2023-11-24T06:21:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 -- 172.21.15.89:0/2918174619 learned_addr learned my addr 172.21.15.89:0/2918174619 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 -- 172.21.15.89:0/2918174619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29415e080 msgr2=0x7fb2941f72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29415e080 0x7fb2941f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb298d02700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2940af4d0 0x7fb294153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 -- 172.21.15.89:0/2918174619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2940af4d0 msgr2=0x7fb294153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2940af4d0 0x7fb294153ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 -- 172.21.15.89:0/2918174619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2900094a0 con 0x7fb29406c710 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb299503700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb294160d60 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb28400c550 tx=0x7fb28400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb289ffb700 1 -- 172.21.15.89:0/2918174619 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284004c80 con 0x7fb29406c710 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb289ffb700 1 -- 172.21.15.89:0/2918174619 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb284004e00 con 0x7fb29406c710 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.072+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb294156850 con 0x7fb29406c710 2023-11-24T06:21:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.073+0000 7fb289ffb700 1 -- 172.21.15.89:0/2918174619 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284007310 con 0x7fb29406c710 2023-11-24T06:21:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.073+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb294156e10 con 0x7fb29406c710 2023-11-24T06:21:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.074+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2940753f0 con 0x7fb29406c710 2023-11-24T06:21:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.074+0000 7fb289ffb700 1 -- 172.21.15.89:0/2918174619 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fb284007470 con 0x7fb29406c710 2023-11-24T06:21:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.074+0000 7fb289ffb700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb27806af60 0x7fb27806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.074+0000 7fb289ffb700 1 -- 172.21.15.89:0/2918174619 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb284089300 con 0x7fb29406c710 2023-11-24T06:21:07.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.074+0000 7fb298d02700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb27806af60 0x7fb27806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.075+0000 7fb298d02700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb27806af60 0x7fb27806d420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb2940d02e0 tx=0x7fb290008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.079+0000 7fb289ffb700 1 -- 172.21.15.89:0/2918174619 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2840543c0 con 0x7fb29406c710 2023-11-24T06:21:07.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.312+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb294157030 con 0x7fb29406c710 2023-11-24T06:21:07.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.312+0000 7fb289ffb700 1 -- 172.21.15.89:0/2918174619 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fb2840579e0 con 0x7fb29406c710 2023-11-24T06:21:07.315 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:07.315 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/351443905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":351443905},{"type":"v1","addr":"172.21.15.123:6834","nonce":351443905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6021,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2890850053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2890850053},{"type":"v1","addr":"172.21.15.89:6836","nonce":2890850053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3912248946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3912248946},{"type":"v1","addr":"172.21.15.89:6837","nonce":3912248946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:56.834302+0000","modified":"2023-11-24T06:20:59.846940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6039":{"gid":6039,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2963752713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2963752713},{"type":"v1","addr":"172.21.15.123:6835","nonce":2963752713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb27806af60 msgr2=0x7fb27806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb27806af60 0x7fb27806d420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb2940d02e0 tx=0x7fb290008030 comp rx=0 tx=0).stop 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 msgr2=0x7fb294160d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb294160d60 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb28400c550 tx=0x7fb28400c910 comp rx=0 tx=0).stop 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 shutdown_connections 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29415e080 0x7fb2941f72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29406c710 0x7fb294160d60 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb27806af60 0x7fb27806d420 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 --2- 172.21.15.89:0/2918174619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2940af4d0 0x7fb294153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.314+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 >> 172.21.15.89:0/2918174619 conn(0x7fb294071b90 msgr2=0x7fb29406bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:07.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.315+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 shutdown_connections 2023-11-24T06:21:07.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.315+0000 7fb29b767700 1 -- 172.21.15.89:0/2918174619 wait complete. 2023-11-24T06:21:07.318 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 41 2023-11-24T06:21:07.731 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T06:20:56.834302+0000', 'modified': '2023-11-24T06:20:59.846940+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6039}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6039': {'gid': 6039, 'name': 'b', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6835/2963752713', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6833', 'nonce': 2963752713}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2963752713}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T06:21:07.731 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T06:21:07.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-24T06:21:07.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.855+0000 7f1ecd8ee700 1 Processor -- start 2023-11-24T06:21:07.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.855+0000 7f1ecd8ee700 1 -- start start 2023-11-24T06:21:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ecd8ee700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1ec806c710 0x7f1ec80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ecd8ee700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ec80af4d0 0x7f1ec80afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ecd8ee700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec80ae430 con 0x7f1ec815e080 2023-11-24T06:21:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ecd8ee700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec80af0b0 con 0x7f1ec80af4d0 2023-11-24T06:21:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ecd8ee700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec80b05a0 con 0x7f1ec806c710 2023-11-24T06:21:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ec77fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ec80af4d0 0x7f1ec80afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ec77fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ec80af4d0 0x7f1ec80afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39100/0 (socket says 172.21.15.89:39100) 2023-11-24T06:21:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ec6ffd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1ec815e080 0x7f1ec806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42600/0 (socket says 172.21.15.89:42600) 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ec77fe700 1 -- 172.21.15.89:0/146749931 learned_addr learned my addr 172.21.15.89:0/146749931 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.856+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/146749931 >> v2:172.21.15.123:3300/0 conn(0x7f1ec806c710 0x7f1ec80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ec77fe700 1 -- 172.21.15.89:0/146749931 >> v2:172.21.15.123:3300/0 conn(0x7f1ec806c710 msgr2=0x7f1ec80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ec77fe700 1 --2- 172.21.15.89:0/146749931 >> v2:172.21.15.123:3300/0 conn(0x7f1ec806c710 0x7f1ec80b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ec77fe700 1 -- 172.21.15.89:0/146749931 >> v1:172.21.15.89:6789/0 conn(0x7f1ec815e080 legacy=0x7f1ec806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ec77fe700 1 -- 172.21.15.89:0/146749931 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80ae710 con 0x7f1ec80af4d0 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ec77fe700 1 --2- 172.21.15.89:0/146749931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec80afb90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f1ec806e4e0 tx=0x7f1ec000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a5c792370c101e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ec5ffb700 1 -- 172.21.15.89:0/146749931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000e070 con 0x7f1ec80af4d0 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ec5ffb700 1 -- 172.21.15.89:0/146749931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec000b980 con 0x7f1ec80af4d0 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ec5ffb700 1 -- 172.21.15.89:0/146749931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000bd30 con 0x7f1ec80af4d0 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/146749931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 msgr2=0x7f1ec80afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/146749931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec80afb90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f1ec806e4e0 tx=0x7f1ec000b330 comp rx=0 tx=0).stop 2023-11-24T06:21:07.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/146749931 shutdown_connections 2023-11-24T06:21:07.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/146749931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec80afb90 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/146749931 >> v2:172.21.15.123:3300/0 conn(0x7f1ec806c710 0x7f1ec80b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/146749931 >> 172.21.15.89:0/146749931 conn(0x7f1ec8071b90 msgr2=0x7f1ec8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:07.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/146749931 shutdown_connections 2023-11-24T06:21:07.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.857+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/146749931 wait complete. 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.859+0000 7f1ecd8ee700 1 Processor -- start 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.859+0000 7f1ecd8ee700 1 -- start start 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ecd8ee700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec806c710 0x7f1ec81f7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ecd8ee700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec81572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ecd8ee700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec815e080 0x7f1ec81f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ecd8ee700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec815a5c0 con 0x7f1ec80af4d0 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ecd8ee700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec8157800 con 0x7f1ec815e080 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ecd8ee700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec8159940 con 0x7f1ec806c710 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec81572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec81572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39104/0 (socket says 172.21.15.89:39104) 2023-11-24T06:21:07.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 -- 172.21.15.89:0/728642660 learned_addr learned my addr 172.21.15.89:0/728642660 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec77fe700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec815e080 0x7f1ec81f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec806c710 0x7f1ec81f7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 -- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec806c710 msgr2=0x7f1ec81f7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec806c710 0x7f1ec81f7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 -- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec815e080 msgr2=0x7f1ec81f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec815e080 0x7f1ec81f2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 -- 172.21.15.89:0/728642660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0009b30 con 0x7f1ec80af4d0 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec77fe700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec815e080 0x7f1ec81f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec806c710 0x7f1ec81f7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec81572c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800e810 tx=0x7f1eb800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1eaffff700 1 -- 172.21.15.89:0/728642660 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8013610 con 0x7f1ec80af4d0 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1eaffff700 1 -- 172.21.15.89:0/728642660 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8013770 con 0x7f1ec80af4d0 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/728642660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec8159c20 con 0x7f1ec80af4d0 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1eaffff700 1 -- 172.21.15.89:0/728642660 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8013b20 con 0x7f1ec80af4d0 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.860+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/728642660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec815a1e0 con 0x7f1ec80af4d0 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.862+0000 7f1eaffff700 1 -- 172.21.15.89:0/728642660 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f1eb8013c80 con 0x7f1ec80af4d0 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.863+0000 7f1eaffff700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1eb406af60 0x7f1eb406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.863+0000 7f1eaffff700 1 -- 172.21.15.89:0/728642660 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1eb8089450 con 0x7f1ec80af4d0 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.863+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1eb406af60 0x7f1eb406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.863+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1eb406af60 0x7f1eb406d420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc000f60 tx=0x7f1ebc007df0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.864+0000 7f1ed9ae7b80 1 Processor -- start 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.864+0000 7f1ed9ae7b80 1 -- start start 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.865+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a70080 0x562ee8ab6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.865+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a85ed0 0x562ee8ab6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.865+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a99e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.865+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ee89e3770 con 0x562ee8a85ed0 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.865+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ee89ca750 con 0x562ee8a6d490 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.865+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ee89c8070 con 0x562ee8a70080 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.865+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a85ed0 0x562ee8ab6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.865+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a85ed0 0x562ee8ab6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39112/0 (socket says 172.21.15.89:39112) 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2750214953 learned_addr learned my addr 172.21.15.89:0/2750214953 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a70080 0x562ee8ab6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec77fe700 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a99e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2750214953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a70080 msgr2=0x562ee8ab6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a70080 0x562ee8ab6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2750214953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 msgr2=0x562ee8a99e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a99e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2750214953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ee88405e0 con 0x562ee8a85ed0 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a85ed0 0x562ee8ab6d80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8087970 tx=0x7f1eb8057b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02181940f00f2a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ead7fa700 1 -- 172.21.15.89:0/2750214953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb805c020 con 0x562ee8a85ed0 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ead7fa700 1 -- 172.21.15.89:0/2750214953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb805c220 con 0x562ee8a85ed0 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.866+0000 7f1ead7fa700 1 -- 172.21.15.89:0/2750214953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb80603d0 con 0x562ee8a85ed0 2023-11-24T06:21:07.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2750214953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a85ed0 msgr2=0x562ee8ab6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a85ed0 0x562ee8ab6d80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8087970 tx=0x7f1eb8057b00 comp rx=0 tx=0).stop 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2750214953 shutdown_connections 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a99e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a70080 0x562ee8ab6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2750214953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a85ed0 0x562ee8ab6d80 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2750214953 >> 172.21.15.89:0/2750214953 conn(0x562ee899e310 msgr2=0x562ee8ab4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2750214953 shutdown_connections 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2750214953 wait complete. 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.867+0000 7f1ed9ae7b80 1 -- start start 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ed9ae7b80 1 Processor -- start 2023-11-24T06:21:07.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8aadea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8aae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8a64330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ee89c3960 con 0x562ee8a70080 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ee897d510 con 0x562ee8a6d490 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ee897d1b0 con 0x562ee8a85ed0 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ed9ae7b80 10 client.0 _ll_register_callbacks cb 0x562ee88a8a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8aae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8aae3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39118/0 (socket says 172.21.15.89:39118) 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2434239796 learned_addr learned my addr 172.21.15.89:0/2434239796 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ec77fe700 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8a64330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.868+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8aadea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 msgr2=0x562ee8a64330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8a64330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 msgr2=0x562ee8aadea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8aadea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2434239796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec815a1e0 con 0x562ee8a70080 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8aae3e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8058730 tx=0x7f1eb8089070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a971b05a2bf991 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ea67fc700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ea67fc700 1 -- 172.21.15.89:0/2434239796 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb80603d0 con 0x562ee8a70080 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ea67fc700 1 -- 172.21.15.89:0/2434239796 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1eb80575c0 con 0x562ee8a70080 2023-11-24T06:21:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2434239796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ee8a62a40 con 0x562ee8a70080 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ea67fc700 1 -- 172.21.15.89:0/2434239796 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1eb8057780 con 0x562ee8a70080 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 10 client.6207 fetch_fsmap learned FSMap version 41 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2434239796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x562ee87ecd90 con 0x562ee8a70080 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ea67fc700 1 -- 172.21.15.89:0/2434239796 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562ee8a62a40 con 0x562ee8a70080 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ea67fc700 1 -- 172.21.15.89:0/2434239796 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 41) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1eb805ff40 con 0x562ee8a70080 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 10 client.6207 fetch_fsmap finished waiting for FSMap version 41 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 10 client.6207 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 20 client.6207 populate_metadata read hostname 'smithi089' 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] conn(0x562ee8a9c6f0 0x562ee8a9cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 4 client.6207 mds_command: new command op to 6039 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T06:21:07.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.869+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2434239796 --> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ee8a3e840 con 0x562ee8a9c6f0 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.870+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] conn(0x562ee8a9c6f0 0x562ee8a9cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.870+0000 7f1ea67fc700 10 client.6207 ms_handle_connect on v2:172.21.15.123:6833/2963752713 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.870+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] conn(0x562ee8a9c6f0 0x562ee8a9cad0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3703046c726155e server_cookie=fbe2340590083f2 in_seq=0 out_seq=0 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ea67fc700 1 -- 172.21.15.89:0/2434239796 <== mds.0 v2:172.21.15.123:6833/2963752713 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562ee8a3e840 con 0x562ee8a9c6f0 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ea67fc700 10 client.6207 handle_command_reply: tid=0 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 client.6207 shutdown 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2434239796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 msgr2=0x562ee8aae3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8aae3e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8058730 tx=0x7f1eb8089070 comp rx=0 tx=0).stop 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2434239796 shutdown_connections 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8a64330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8aadea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] conn(0x562ee8a9c6f0 0x562ee8a9cad0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2434239796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8aae3e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ea67fc700 0 client.6207 ms_handle_reset on v2:172.21.15.123:6833/2963752713 2023-11-24T06:21:07.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2434239796 >> 172.21.15.89:0/2434239796 conn(0x562ee899e310 msgr2=0x562ee8ab8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:07.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2434239796 shutdown_connections 2023-11-24T06:21:07.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2434239796 wait complete. 2023-11-24T06:21:07.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.871+0000 7f1ed9ae7b80 20 client.6207 trim_cache size 0 max 16384 2023-11-24T06:21:07.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.889+0000 7f1ed9ae7b80 1 Processor -- start 2023-11-24T06:21:07.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.889+0000 7f1ed9ae7b80 1 -- start start 2023-11-24T06:21:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a6c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ac3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8a63cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ee88428d0 con 0x562ee8a70080 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ee8aabb50 con 0x562ee8a6d490 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ee8a6ce20 con 0x562ee8a85ed0 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ac3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ac3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39126/0 (socket says 172.21.15.89:39126) 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2839346640 learned_addr learned my addr 172.21.15.89:0/2839346640 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2839346640 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 msgr2=0x562ee8a63cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8a63cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2839346640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 msgr2=0x562ee8a6c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a6c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a6c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 -- 172.21.15.89:0/2839346640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ee87d5c10 con 0x562ee8a70080 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a6c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ac3970 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8057a50 tx=0x7f1eb805a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a84710c702fc5b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1eadffb700 1 -- 172.21.15.89:0/2839346640 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8087860 con 0x562ee8a70080 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1eadffb700 1 -- 172.21.15.89:0/2839346640 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb80603d0 con 0x562ee8a70080 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.890+0000 7f1eadffb700 1 -- 172.21.15.89:0/2839346640 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb80575c0 con 0x562ee8a70080 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2839346640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 msgr2=0x562ee8ac3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ac3970 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8057a50 tx=0x7f1eb805a000 comp rx=0 tx=0).stop 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2839346640 shutdown_connections 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8a63cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a6c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/2839346640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ac3970 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2839346640 >> 172.21.15.89:0/2839346640 conn(0x562ee899e310 msgr2=0x562ee8aaea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2839346640 shutdown_connections 2023-11-24T06:21:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.891+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/2839346640 wait complete. 2023-11-24T06:21:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.892+0000 7f1ed9ae7b80 1 -- start start 2023-11-24T06:21:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.892+0000 7f1ed9ae7b80 1 Processor -- start 2023-11-24T06:21:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.892+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a9b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ab2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8acdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ee8ab3550 con 0x562ee8a70080 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ee8aab610 con 0x562ee8a6d490 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ee8acd950 con 0x562ee8a85ed0 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 10 client.0 _ll_register_callbacks cb 0x562ee88a8a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ab2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ab2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39132/0 (socket says 172.21.15.89:39132) 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 -- 172.21.15.89:0/1542244867 learned_addr learned my addr 172.21.15.89:0/1542244867 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a9b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec77fe700 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8acdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 -- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 msgr2=0x562ee8acdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8acdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 -- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 msgr2=0x562ee8a9b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a9b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 -- 172.21.15.89:0/1542244867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ee87ecd90 con 0x562ee8a70080 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec77fe700 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8acdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ec67fc700 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ab2ce0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8057b20 tx=0x7f1eb8054490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b46128b3dd7fc6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ea67fc700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ea67fc700 1 -- 172.21.15.89:0/1542244867 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8075120 con 0x562ee8a70080 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ea67fc700 1 -- 172.21.15.89:0/1542244867 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1eb8075280 con 0x562ee8a70080 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/1542244867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1eb805c220 con 0x562ee8a70080 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ea67fc700 1 -- 172.21.15.89:0/1542244867 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1eb8075440 con 0x562ee8a70080 2023-11-24T06:21:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 10 client.6213 fetch_fsmap learned FSMap version 41 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/1542244867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x562ee8813200 con 0x562ee8a70080 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ea67fc700 1 -- 172.21.15.89:0/1542244867 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb805c220 con 0x562ee8a70080 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ea67fc700 1 -- 172.21.15.89:0/1542244867 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 41) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1eb8087ae0 con 0x562ee8a70080 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.893+0000 7f1ed9ae7b80 10 client.6213 fetch_fsmap finished waiting for FSMap version 41 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ed9ae7b80 10 client.6213 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ed9ae7b80 20 client.6213 populate_metadata read hostname 'smithi089' 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] conn(0x562ee8ace370 0x562ee8aaa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ed9ae7b80 4 client.6213 mds_command: new command op to 6039 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/1542244867 --> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562ee8989f80 con 0x562ee8ace370 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] conn(0x562ee8ace370 0x562ee8aaa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ea67fc700 10 client.6213 ms_handle_connect on v2:172.21.15.123:6833/2963752713 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ec6ffd700 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] conn(0x562ee8ace370 0x562ee8aaa800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a92d4e2e961d1faa server_cookie=f737bf9afd2d8386 in_seq=0 out_seq=0 2023-11-24T06:21:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ea67fc700 1 -- 172.21.15.89:0/1542244867 <== mds.0 v2:172.21.15.123:6833/2963752713 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x562ee8989f80 con 0x562ee8ace370 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ea67fc700 10 client.6213 handle_command_reply: tid=0 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.894+0000 7f1ed9ae7b80 1 client.6213 shutdown 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/1542244867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 msgr2=0x562ee8ab2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ab2ce0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8057b20 tx=0x7f1eb8054490 comp rx=0 tx=0).stop 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/1542244867 shutdown_connections 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x562ee8a85ed0 0x562ee8acdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562ee8a6d490 0x562ee8a9b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.123:6833/2963752713,v1:172.21.15.123:6835/2963752713] conn(0x562ee8ace370 0x562ee8aaa800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 --2- 172.21.15.89:0/1542244867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562ee8a70080 0x562ee8ab2ce0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ea67fc700 0 client.6213 ms_handle_reset on v2:172.21.15.123:6833/2963752713 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/1542244867 >> 172.21.15.89:0/1542244867 conn(0x562ee899e310 msgr2=0x562ee8ac3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/1542244867 shutdown_connections 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 1 -- 172.21.15.89:0/1542244867 wait complete. 2023-11-24T06:21:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ed9ae7b80 20 client.6213 trim_cache size 0 max 16384 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "be2488f4-68a7-42cc-93d1-ae9c6921fb48", 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 6039, 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 10.460337561999999, 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 41, 2023-11-24T06:21:07.905 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 64, 2023-11-24T06:21:07.906 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 64, 2023-11-24T06:21:07.906 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 15.396638209000001 2023-11-24T06:21:07.906 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1eb406af60 msgr2=0x7f1eb406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1eb406af60 0x7f1eb406d420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc000f60 tx=0x7f1ebc007df0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 msgr2=0x7f1ec81572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec81572c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800e810 tx=0x7f1eb800ebd0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/728642660 shutdown_connections 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1eb406af60 0x7f1eb406d420 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec815e080 0x7f1ec81f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec806c710 0x7f1ec81f7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 --2- 172.21.15.89:0/728642660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ec80af4d0 0x7f1ec81572c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/728642660 >> 172.21.15.89:0/728642660 conn(0x7f1ec8071b90 msgr2=0x7f1ec806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/728642660 shutdown_connections 2023-11-24T06:21:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:07.895+0000 7f1ecd8ee700 1 -- 172.21.15.89:0/728642660 wait complete. 2023-11-24T06:21:07.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:21:08.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.029+0000 7efd8541e700 1 Processor -- start 2023-11-24T06:21:08.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.029+0000 7efd8541e700 1 -- start start 2023-11-24T06:21:08.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd8541e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800b0fb0 0x7efd800cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:08.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd8541e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efd800b1390 0x7efd8006b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:08.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd8541e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd800d0280 0x7efd8006bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:08.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd8541e700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd800b1c60 con 0x7efd800b1390 2023-11-24T06:21:08.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd8541e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd800d0720 con 0x7efd800d0280 2023-11-24T06:21:08.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd8541e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd800d0a00 con 0x7efd800b0fb0 2023-11-24T06:21:08.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efd800b1390 0x7efd8006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:08.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efd800b1390 0x7efd8006b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39140/0 (socket says 172.21.15.89:39140) 2023-11-24T06:21:08.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 -- 172.21.15.89:0/2078545140 learned_addr learned my addr 172.21.15.89:0/2078545140 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:08.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7f7fe700 1 --2- 172.21.15.89:0/2078545140 >> v2:172.21.15.123:3300/0 conn(0x7efd800d0280 0x7efd8006bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:08.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7effd700 1 --2- 172.21.15.89:0/2078545140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800b0fb0 0x7efd800cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:08.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 -- 172.21.15.89:0/2078545140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800b0fb0 msgr2=0x7efd800cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:08.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 --2- 172.21.15.89:0/2078545140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800b0fb0 0x7efd800cfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 -- 172.21.15.89:0/2078545140 >> v2:172.21.15.123:3300/0 conn(0x7efd800d0280 msgr2=0x7efd8006bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:08.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 --2- 172.21.15.89:0/2078545140 >> v2:172.21.15.123:3300/0 conn(0x7efd800d0280 0x7efd8006bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 -- 172.21.15.89:0/2078545140 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd800d0ce0 con 0x7efd800b1390 2023-11-24T06:21:08.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7e7fc700 1 --2- 172.21.15.89:0/2078545140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd8006b750 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7efd800c8d00 tx=0x7efd74009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a777fb9ace718321 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:08.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7dffb700 1 -- 172.21.15.89:0/2078545140 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7400b870 con 0x7efd800b1390 2023-11-24T06:21:08.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7dffb700 1 -- 172.21.15.89:0/2078545140 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7400b9d0 con 0x7efd800b1390 2023-11-24T06:21:08.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.030+0000 7efd7dffb700 1 -- 172.21.15.89:0/2078545140 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7400bd60 con 0x7efd800b1390 2023-11-24T06:21:08.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 -- 172.21.15.89:0/2078545140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 msgr2=0x7efd8006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:08.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 --2- 172.21.15.89:0/2078545140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd8006b750 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7efd800c8d00 tx=0x7efd74009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 -- 172.21.15.89:0/2078545140 shutdown_connections 2023-11-24T06:21:08.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 --2- 172.21.15.89:0/2078545140 >> v2:172.21.15.123:3300/0 conn(0x7efd800d0280 0x7efd8006bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 --2- 172.21.15.89:0/2078545140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800b0fb0 0x7efd800cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 --2- 172.21.15.89:0/2078545140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd8006b750 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 -- 172.21.15.89:0/2078545140 >> 172.21.15.89:0/2078545140 conn(0x7efd8009aa00 msgr2=0x7efd8009ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:08.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 -- 172.21.15.89:0/2078545140 shutdown_connections 2023-11-24T06:21:08.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.031+0000 7efd8541e700 1 -- 172.21.15.89:0/2078545140 wait complete. 2023-11-24T06:21:08.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.033+0000 7efd8541e700 1 Processor -- start 2023-11-24T06:21:08.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.033+0000 7efd8541e700 1 -- start start 2023-11-24T06:21:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.033+0000 7efd8541e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd800b0fb0 0x7efd800c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd8541e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd800b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd8541e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800d0280 0x7efd8006fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd8541e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd800c6440 con 0x7efd800b1390 2023-11-24T06:21:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd8541e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd800c57c0 con 0x7efd800b0fb0 2023-11-24T06:21:08.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd8541e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd800c5aa0 con 0x7efd800d0280 2023-11-24T06:21:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd800b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd800b66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39146/0 (socket says 172.21.15.89:39146) 2023-11-24T06:21:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 -- 172.21.15.89:0/3640864840 learned_addr learned my addr 172.21.15.89:0/3640864840 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7effd700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd800b0fb0 0x7efd800c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7f7fe700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800d0280 0x7efd8006fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 -- 172.21.15.89:0/3640864840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800d0280 msgr2=0x7efd8006fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:08.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800d0280 0x7efd8006fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 -- 172.21.15.89:0/3640864840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd800b0fb0 msgr2=0x7efd800c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:08.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd800b0fb0 0x7efd800c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 -- 172.21.15.89:0/3640864840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74009550 con 0x7efd800b1390 2023-11-24T06:21:08.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd7e7fc700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd800b66a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7efd74000c00 tx=0x7efd74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:08.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd677fe700 1 -- 172.21.15.89:0/3640864840 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74004480 con 0x7efd800b1390 2023-11-24T06:21:08.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd677fe700 1 -- 172.21.15.89:0/3640864840 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd740045e0 con 0x7efd800b1390 2023-11-24T06:21:08.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd800c5d80 con 0x7efd800b1390 2023-11-24T06:21:08.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd677fe700 1 -- 172.21.15.89:0/3640864840 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd740038e0 con 0x7efd800b1390 2023-11-24T06:21:08.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.034+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd800b6be0 con 0x7efd800b1390 2023-11-24T06:21:08.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.035+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd80068480 con 0x7efd800b1390 2023-11-24T06:21:08.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.036+0000 7efd677fe700 1 -- 172.21.15.89:0/3640864840 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7efd74021070 con 0x7efd800b1390 2023-11-24T06:21:08.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.036+0000 7efd677fe700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd6c06af60 0x7efd6c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:08.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.036+0000 7efd677fe700 1 -- 172.21.15.89:0/3640864840 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd7408ac40 con 0x7efd800b1390 2023-11-24T06:21:08.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.036+0000 7efd7effd700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd6c06af60 0x7efd6c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:08.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.037+0000 7efd7effd700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd6c06af60 0x7efd6c06d420 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efd70009950 tx=0x7efd70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:08.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.041+0000 7efd677fe700 1 -- 172.21.15.89:0/3640864840 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd74055db0 con 0x7efd800b1390 2023-11-24T06:21:08.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.275+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efd800b6d70 con 0x7efd800b1390 2023-11-24T06:21:08.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.275+0000 7efd677fe700 1 -- 172.21.15.89:0/3640864840 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7efd74015020 con 0x7efd800b1390 2023-11-24T06:21:08.277 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:21:08.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd6c06af60 msgr2=0x7efd6c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:08.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd6c06af60 0x7efd6c06d420 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efd70009950 tx=0x7efd70008040 comp rx=0 tx=0).stop 2023-11-24T06:21:08.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 msgr2=0x7efd800b66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:08.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd800b66a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7efd74000c00 tx=0x7efd74004750 comp rx=0 tx=0).stop 2023-11-24T06:21:08.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 shutdown_connections 2023-11-24T06:21:08.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd800d0280 0x7efd8006fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd6c06af60 0x7efd6c06d420 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd800b0fb0 0x7efd800c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 --2- 172.21.15.89:0/3640864840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd800b1390 0x7efd800b66a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:08.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 >> 172.21.15.89:0/3640864840 conn(0x7efd8009aa00 msgr2=0x7efd800b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:08.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 shutdown_connections 2023-11-24T06:21:08.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:08.277+0000 7efd8541e700 1 -- 172.21.15.89:0/3640864840 wait complete. 2023-11-24T06:21:08.291 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T06:21:08.291 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:21:08.292 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2023-11-24T06:21:08.292 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T06:21:08.292 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T06:21:08.292 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T06:21:08.293 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T06:21:08.293 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:21:08.294 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:21:08.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.294 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2023-11-24T06:21:08.312 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:21:08.312 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:21:08.313 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:21:08.313 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:08.313 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-11-24T06:21:08.314 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:08.314 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:21:08.314 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:08.314 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:21:08.314 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:08.315 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:21:08.315 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:08.315 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:21:08.315 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:08.315 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:21:08.315 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42293sec preferred_lft 42293sec 2023-11-24T06:21:08.315 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4376:7845:d8a4:7bc1/64 scope link noprefixroute 2023-11-24T06:21:08.316 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:08.316 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:21:08.316 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 12:de:17:7b:11:37 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:08.316 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:21:08.316 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:08.316 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::3cfa:7aff:fed2:90c8/64 scope link 2023-11-24T06:21:08.316 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:08.316 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:21:08.317 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 12:de:17:7b:11:37 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:21:08.317 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::10de:17ff:fe7b:1137/64 scope link 2023-11-24T06:21:08.317 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:08.317 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:21:08.317 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 76:9a:20:1a:a1:88 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T06:21:08.317 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::749a:20ff:fe1a:a188/64 scope link 2023-11-24T06:21:08.317 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:08.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.318 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:21:08.380 INFO:teuthology.orchestra.run.smithi089.stdout:1 2023-11-24T06:21:08.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.381 DEBUG:teuthology.orchestra.run.smithi089:> route 2023-11-24T06:21:08.440 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2023-11-24T06:21:08.441 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:21:08.441 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:21:08.441 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:21:08.441 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:21:08.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.441 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-11-24T06:21:08.441 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-11-24T06:21:08.441 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:21:08.441 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:21:08.441 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:21:08.442 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-11-24T06:21:08.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.545 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:21:08.561 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T06:21:08.562 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T06:21:08.562 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:21:08.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2023-11-24T06:21:08.628 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:21:08.628 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T06:21:08.684 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:21:08.685 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:21:08.685 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:21:08.741 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:21:08.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.741 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:21:08.826 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:21:08.826 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:21:08.826 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:21:08.835 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:21:08.835 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:21:08.835 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:21:08.835 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:21:08.835 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:21:08.866 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-24T06:21:08.866 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-24T06:21:08.866 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-11-24T06:21:08.866 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:21:08.866 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:21:08.867 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:21:08.867 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:21:08.867 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:21:08.876 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:21:08.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:21:08.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:08.908 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:21:08.997 INFO:teuthology.orchestra.run.smithi089.stdout:client5297 2023-11-24T06:21:08.997 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:21:08.997 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:09.014 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:21:09.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.173+0000 7fed958fe700 1 Processor -- start 2023-11-24T06:21:09.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.174+0000 7fed958fe700 1 -- start start 2023-11-24T06:21:09.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.174+0000 7fed958fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed9015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:09.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.174+0000 7fed958fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed900b2a40 0x7fed90070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:09.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.174+0000 7fed958fe700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed900b30b0 con 0x7fed901587e0 2023-11-24T06:21:09.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.174+0000 7fed958fe700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed900b3d30 con 0x7fed900b2a40 2023-11-24T06:21:09.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.174+0000 7fed958fe700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed900b2eb0 con 0x7fed9006eb70 2023-11-24T06:21:09.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed900b2a40 0x7fed90070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:09.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8f7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed900b2a40 0x7fed90070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39160/0 (socket says 172.21.15.89:39160) 2023-11-24T06:21:09.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8effd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fed901587e0 0x7fed9006e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42660/0 (socket says 172.21.15.89:42660) 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8f7fe700 1 -- 172.21.15.89:0/3636792261 learned_addr learned my addr 172.21.15.89:0/3636792261 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8e7fc700 1 --2- 172.21.15.89:0/3636792261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed9015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8f7fe700 1 -- 172.21.15.89:0/3636792261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 msgr2=0x7fed9015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8f7fe700 1 --2- 172.21.15.89:0/3636792261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed9015b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8f7fe700 1 -- 172.21.15.89:0/3636792261 >> v1:172.21.15.89:6789/0 conn(0x7fed901587e0 legacy=0x7fed9006e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8dffb700 1 -- 172.21.15.89:0/3636792261 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120202845 0 0) 0x7fed900b30b0 con 0x7fed901587e0 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8f7fe700 1 -- 172.21.15.89:0/3636792261 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed900b3390 con 0x7fed900b2a40 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8e7fc700 1 --2- 172.21.15.89:0/3636792261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed9015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8f7fe700 1 --2- 172.21.15.89:0/3636792261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed90070c70 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fed88009e50 tx=0x7fed8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90425ed4a9b88671 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8dffb700 1 -- 172.21.15.89:0/3636792261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8800e070 con 0x7fed900b2a40 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8dffb700 1 -- 172.21.15.89:0/3636792261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8800b9c0 con 0x7fed900b2a40 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.175+0000 7fed8dffb700 1 -- 172.21.15.89:0/3636792261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8800bd70 con 0x7fed900b2a40 2023-11-24T06:21:09.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.176+0000 7fed958fe700 1 -- 172.21.15.89:0/3636792261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 msgr2=0x7fed90070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.176+0000 7fed958fe700 1 --2- 172.21.15.89:0/3636792261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed90070c70 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fed88009e50 tx=0x7fed8800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.176+0000 7fed958fe700 1 -- 172.21.15.89:0/3636792261 shutdown_connections 2023-11-24T06:21:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.176+0000 7fed958fe700 1 --2- 172.21.15.89:0/3636792261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed90070c70 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.176+0000 7fed958fe700 1 --2- 172.21.15.89:0/3636792261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed9015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.176+0000 7fed958fe700 1 -- 172.21.15.89:0/3636792261 >> 172.21.15.89:0/3636792261 conn(0x7fed900b1740 msgr2=0x7fed900b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.176+0000 7fed958fe700 1 -- 172.21.15.89:0/3636792261 shutdown_connections 2023-11-24T06:21:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.176+0000 7fed958fe700 1 -- 172.21.15.89:0/3636792261 wait complete. 2023-11-24T06:21:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.177+0000 7fed958fe700 1 Processor -- start 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.177+0000 7fed958fe700 1 -- start start 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed958fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed901444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed958fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed9006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed958fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed901587e0 0x7fed9006d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed958fe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed901449e0 con 0x7fed900b2a40 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed958fe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed900cdbf0 con 0x7fed901587e0 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed958fe700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed900cded0 con 0x7fed9006eb70 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed9006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed9006d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39164/0 (socket says 172.21.15.89:39164) 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 -- 172.21.15.89:0/2588911860 learned_addr learned my addr 172.21.15.89:0/2588911860 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8effd700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed901444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 -- 172.21.15.89:0/2588911860 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 msgr2=0x7fed901444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed901444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 -- 172.21.15.89:0/2588911860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed901587e0 msgr2=0x7fed9006d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed901587e0 0x7fed9006d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 -- 172.21.15.89:0/2588911860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88009b30 con 0x7fed900b2a40 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.178+0000 7fed8e7fc700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed9006d350 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fed8000c750 tx=0x7fed8000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.179+0000 7fed77fff700 1 -- 172.21.15.89:0/2588911860 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80011800 con 0x7fed900b2a40 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.179+0000 7fed77fff700 1 -- 172.21.15.89:0/2588911860 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80011960 con 0x7fed900b2a40 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.179+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed900ce1b0 con 0x7fed900b2a40 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.179+0000 7fed77fff700 1 -- 172.21.15.89:0/2588911860 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80011cd0 con 0x7fed900b2a40 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.179+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed900ce770 con 0x7fed900b2a40 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.180+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed900751e0 con 0x7fed900b2a40 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.180+0000 7fed77fff700 1 -- 172.21.15.89:0/2588911860 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fed80021350 con 0x7fed900b2a40 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.180+0000 7fed77fff700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed7c06af60 0x7fed7c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.181+0000 7fed8effd700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed7c06af60 0x7fed7c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.181+0000 7fed77fff700 1 -- 172.21.15.89:0/2588911860 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed80089690 con 0x7fed900b2a40 2023-11-24T06:21:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.181+0000 7fed8effd700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed7c06af60 0x7fed7c06d420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fed84000f60 tx=0x7fed84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.185+0000 7fed77fff700 1 -- 172.21.15.89:0/2588911860 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed800546d0 con 0x7fed900b2a40 2023-11-24T06:21:09.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.420+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fed9006ceb0 con 0x7fed900b2a40 2023-11-24T06:21:09.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.420+0000 7fed77fff700 1 -- 172.21.15.89:0/2588911860 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7fed80057cf0 con 0x7fed900b2a40 2023-11-24T06:21:09.423 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:21:09.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed7c06af60 msgr2=0x7fed7c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed7c06af60 0x7fed7c06d420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fed84000f60 tx=0x7fed84009040 comp rx=0 tx=0).stop 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 msgr2=0x7fed9006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed9006d350 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fed8000c750 tx=0x7fed8000cb10 comp rx=0 tx=0).stop 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 shutdown_connections 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed7c06af60 0x7fed7c06d420 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed901587e0 0x7fed9006d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed9006eb70 0x7fed901444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 --2- 172.21.15.89:0/2588911860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed900b2a40 0x7fed9006d350 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:09.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 >> 172.21.15.89:0/2588911860 conn(0x7fed900b1740 msgr2=0x7fed9015baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:09.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 shutdown_connections 2023-11-24T06:21:09.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:09.422+0000 7fed958fe700 1 -- 172.21.15.89:0/2588911860 wait complete. 2023-11-24T06:21:09.436 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T06:21:09.437 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:21:09.437 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2023-11-24T06:21:09.437 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T06:21:09.437 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T06:21:09.437 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T06:21:09.437 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T06:21:09.437 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:21:09.437 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:21:09.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:09.438 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2023-11-24T06:21:09.459 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:21:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:21:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:21:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:09.459 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42282sec preferred_lft 42282sec 2023-11-24T06:21:09.460 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::a981:dd7b:bd0d:851a/64 scope link noprefixroute 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 1a:69:ea:ee:dc:d5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::682d:1bff:fea0:f45c/64 scope link 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether d2:06:87:3a:6b:fb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::d006:87ff:fe3a:6bfb/64 scope link 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:09.461 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:21:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 1a:69:ea:ee:dc:d5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T06:21:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::1869:eaff:feee:dcd5/64 scope link 2023-11-24T06:21:09.462 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:21:09.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:09.462 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:21:09.531 INFO:teuthology.orchestra.run.smithi123.stdout:1 2023-11-24T06:21:09.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:09.532 DEBUG:teuthology.orchestra.run.smithi123:> route 2023-11-24T06:21:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2023-11-24T06:21:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:21:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:21:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:21:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:21:09.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:09.596 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2023-11-24T06:21:09.596 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2023-11-24T06:21:09.596 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:21:09.596 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:21:09.596 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:21:09.596 DEBUG:teuthology.orchestra.run.smithi123:> ') 2023-11-24T06:21:09.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:09.716 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:21:09.736 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T06:21:09.737 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T06:21:09.737 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:21:09.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:09.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2023-11-24T06:21:09.809 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:21:09.809 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T06:21:09.870 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:21:09.871 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:21:09.871 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:21:09.930 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-24T06:21:09.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:09.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:21:10.026 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:21:10.026 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:21:10.027 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:21:10.036 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:21:10.036 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:21:10.036 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:21:10.037 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:21:10.037 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:21:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-24T06:21:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-24T06:21:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:invalid new device string format 2023-11-24T06:21:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.123:3300,172.21.15.123:3301" 2023-11-24T06:21:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:21:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790" 2023-11-24T06:21:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.123:6789,172.21.15.123:6790:/ 2023-11-24T06:21:10.072 INFO:teuthology.orchestra.run.smithi123.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=be2488f4-68a7-42cc-93d1-ae9c6921fb48" will pass to kernel 2023-11-24T06:21:10.075 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:21:10.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:10.076 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:21:10.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:10.110 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:21:10.183 INFO:teuthology.orchestra.run.smithi123.stdout:client5300 2023-11-24T06:21:10.183 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:21:10.184 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:10.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T06:21:10.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.371+0000 7f1c62ab7700 1 Processor -- start 2023-11-24T06:21:10.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.372+0000 7f1c62ab7700 1 -- start start 2023-11-24T06:21:10.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.372+0000 7f1c62ab7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c5c0b01e0 0x7f1c5c06e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:10.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.372+0000 7f1c62ab7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 0x7f1c5c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:10.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.372+0000 7f1c62ab7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c0b05c0 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.372+0000 7f1c62ab7700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c5c1470a0 con 0x7f1c5c0b01e0 2023-11-24T06:21:10.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.372+0000 7f1c62ab7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c0b5e70 con 0x7f1c5c0746a0 2023-11-24T06:21:10.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c60853700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c5c0b01e0 0x7f1c5c06e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c60853700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c5c0b01e0 0x7f1c5c06e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39170/0 (socket says 172.21.15.89:39170) 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c60853700 1 -- 172.21.15.89:0/4128243699 learned_addr learned my addr 172.21.15.89:0/4128243699 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c61054700 1 --2- 172.21.15.89:0/4128243699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 0x7f1c5c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c60853700 1 -- 172.21.15.89:0/4128243699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 msgr2=0x7f1c5c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c60853700 1 --2- 172.21.15.89:0/4128243699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 0x7f1c5c0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c60853700 1 -- 172.21.15.89:0/4128243699 >> v1:172.21.15.123:6789/0 conn(0x7f1c5c0b59c0 legacy=0x7f1c5c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c60853700 1 -- 172.21.15.89:0/4128243699 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c0b4250 con 0x7f1c5c0b01e0 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c60853700 1 --2- 172.21.15.89:0/4128243699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b01e0 0x7f1c5c06e9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009920 tx=0x7f1c50009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18413e1e65a5ead9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c5b7fe700 1 -- 172.21.15.89:0/4128243699 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50011830 con 0x7f1c5c0b01e0 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c5b7fe700 1 -- 172.21.15.89:0/4128243699 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c50011990 con 0x7f1c5c0b01e0 2023-11-24T06:21:10.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.373+0000 7f1c5b7fe700 1 -- 172.21.15.89:0/4128243699 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50011d20 con 0x7f1c5c0b01e0 2023-11-24T06:21:10.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.374+0000 7f1c62ab7700 1 -- 172.21.15.89:0/4128243699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b01e0 msgr2=0x7f1c5c06e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:10.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.374+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/4128243699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b01e0 0x7f1c5c06e9c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009920 tx=0x7f1c50009c30 comp rx=0 tx=0).stop 2023-11-24T06:21:10.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.374+0000 7f1c62ab7700 1 -- 172.21.15.89:0/4128243699 shutdown_connections 2023-11-24T06:21:10.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.374+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/4128243699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 0x7f1c5c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:10.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.374+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/4128243699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b01e0 0x7f1c5c06e9c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:10.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.374+0000 7f1c62ab7700 1 -- 172.21.15.89:0/4128243699 >> 172.21.15.89:0/4128243699 conn(0x7f1c5c0d0b40 msgr2=0x7f1c5c0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:10.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.374+0000 7f1c62ab7700 1 -- 172.21.15.89:0/4128243699 shutdown_connections 2023-11-24T06:21:10.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.374+0000 7f1c62ab7700 1 -- 172.21.15.89:0/4128243699 wait complete. 2023-11-24T06:21:10.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.375+0000 7f1c62ab7700 1 Processor -- start 2023-11-24T06:21:10.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.375+0000 7f1c62ab7700 1 -- start start 2023-11-24T06:21:10.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c62ab7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 0x7f1c5c1f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:10.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c62ab7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5c0b01e0 0x7f1c5c160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:10.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c62ab7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b59c0 0x7f1c5c1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:10.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c62ab7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c1f1320 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c62ab7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c1606d0 con 0x7f1c5c0b01e0 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c62ab7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c15fac0 con 0x7f1c5c0746a0 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b59c0 0x7f1c5c1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b59c0 0x7f1c5c1610a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39178/0 (socket says 172.21.15.89:39178) 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 -- 172.21.15.89:0/1046292275 learned_addr learned my addr 172.21.15.89:0/1046292275 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c60853700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 0x7f1c5c1f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c5bfff700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5c0b01e0 0x7f1c5c160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 -- 172.21.15.89:0/1046292275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 msgr2=0x7f1c5c1f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 0x7f1c5c1f4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 -- 172.21.15.89:0/1046292275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5c0b01e0 msgr2=0x7f1c5c160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5c0b01e0 0x7f1c5c160b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:10.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 -- 172.21.15.89:0/1046292275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c50009550 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.376+0000 7f1c61054700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b59c0 0x7f1c5c1610a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400efc0 tx=0x7f1c5400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.377+0000 7f1c597fa700 1 -- 172.21.15.89:0/1046292275 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c54009ea0 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.377+0000 7f1c597fa700 1 -- 172.21.15.89:0/1046292275 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5400f040 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.377+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5c15fda0 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.377+0000 7f1c597fa700 1 -- 172.21.15.89:0/1046292275 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5400ced0 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.377+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5c160360 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.378+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5c160840 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.378+0000 7f1c597fa700 1 -- 172.21.15.89:0/1046292275 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f1c54014480 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.378+0000 7f1c597fa700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1c4006af60 0x7f1c4006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.379+0000 7f1c60853700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1c4006af60 0x7f1c4006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.379+0000 7f1c597fa700 1 -- 172.21.15.89:0/1046292275 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c54089ee0 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.379+0000 7f1c60853700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1c4006af60 0x7f1c4006d420 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1c50000c00 tx=0x7f1c50008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:10.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.383+0000 7f1c597fa700 1 -- 172.21.15.89:0/1046292275 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c54054e90 con 0x7f1c5c0b59c0 2023-11-24T06:21:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:10.619+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1c5c067f60 con 0x7f1c5c0b59c0 2023-11-24T06:21:11.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.015+0000 7f1c597fa700 1 -- 172.21.15.89:0/1046292275 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 136+0+0 (secure 0 0 0) 0x7f1c540584b0 con 0x7f1c5c0b59c0 2023-11-24T06:21:11.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1c4006af60 msgr2=0x7f1c4006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1c4006af60 0x7f1c4006d420 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1c50000c00 tx=0x7f1c50008030 comp rx=0 tx=0).stop 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b59c0 msgr2=0x7f1c5c1610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b59c0 0x7f1c5c1610a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400efc0 tx=0x7f1c5400c3a0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 shutdown_connections 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c5c0b59c0 0x7f1c5c1610a0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c5c0746a0 0x7f1c5c1f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f1c4006af60 0x7f1c4006d420 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 --2- 172.21.15.89:0/1046292275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5c0b01e0 0x7f1c5c160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 >> 172.21.15.89:0/1046292275 conn(0x7f1c5c0d0b40 msgr2=0x7f1c5c072450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 shutdown_connections 2023-11-24T06:21:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.017+0000 7f1c62ab7700 1 -- 172.21.15.89:0/1046292275 wait complete. 2023-11-24T06:21:11.034 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T06:21:11.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.153+0000 7f7e02b66700 1 Processor -- start 2023-11-24T06:21:11.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.153+0000 7f7e02b66700 1 -- start start 2023-11-24T06:21:11.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e02b66700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7dfc15bdd0 0x7f7dfc1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e02b66700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7dfc0d1c70 0x7f7dfc0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e02b66700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dfc0b22d0 con 0x7f7dfc0b3370 2023-11-24T06:21:11.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e02b66700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dfc0b2f50 con 0x7f7dfc0d1c70 2023-11-24T06:21:11.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e02b66700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dfc0d2640 con 0x7f7dfc15bdd0 2023-11-24T06:21:11.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e01103700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7dfc0d1c70 0x7f7dfc0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e01103700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7dfc0d1c70 0x7f7dfc0d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39186/0 (socket says 172.21.15.89:39186) 2023-11-24T06:21:11.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e01103700 1 -- 172.21.15.89:0/1001700129 learned_addr learned my addr 172.21.15.89:0/1001700129 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:11.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e00902700 1 --2- 172.21.15.89:0/1001700129 >> v2:172.21.15.123:3300/0 conn(0x7f7dfc15bdd0 0x7f7dfc1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e01103700 1 -- 172.21.15.89:0/1001700129 >> v2:172.21.15.123:3300/0 conn(0x7f7dfc15bdd0 msgr2=0x7f7dfc1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:11.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e01103700 1 --2- 172.21.15.89:0/1001700129 >> v2:172.21.15.123:3300/0 conn(0x7f7dfc15bdd0 0x7f7dfc1604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e01103700 1 -- 172.21.15.89:0/1001700129 >> v1:172.21.15.123:6789/0 conn(0x7f7dfc0b3370 legacy=0x7f7dfc0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e01103700 1 -- 172.21.15.89:0/1001700129 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc0b25b0 con 0x7f7dfc0d1c70 2023-11-24T06:21:11.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7e01103700 1 --2- 172.21.15.89:0/1001700129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0d1c70 0x7f7dfc0d2100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc06ec10 tx=0x7f7df400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f359a7133877d72 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7dfb7fe700 1 -- 172.21.15.89:0/1001700129 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df400e070 con 0x7f7dfc0d1c70 2023-11-24T06:21:11.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7dfb7fe700 1 -- 172.21.15.89:0/1001700129 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df400b910 con 0x7f7dfc0d1c70 2023-11-24T06:21:11.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.154+0000 7f7dfb7fe700 1 -- 172.21.15.89:0/1001700129 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df400bcc0 con 0x7f7dfc0d1c70 2023-11-24T06:21:11.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.155+0000 7f7e02b66700 1 -- 172.21.15.89:0/1001700129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0d1c70 msgr2=0x7f7dfc0d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.155+0000 7f7e02b66700 1 --2- 172.21.15.89:0/1001700129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0d1c70 0x7f7dfc0d2100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc06ec10 tx=0x7f7df400b330 comp rx=0 tx=0).stop 2023-11-24T06:21:11.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.155+0000 7f7e02b66700 1 -- 172.21.15.89:0/1001700129 shutdown_connections 2023-11-24T06:21:11.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.155+0000 7f7e02b66700 1 --2- 172.21.15.89:0/1001700129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0d1c70 0x7f7dfc0d2100 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.155+0000 7f7e02b66700 1 --2- 172.21.15.89:0/1001700129 >> v2:172.21.15.123:3300/0 conn(0x7f7dfc15bdd0 0x7f7dfc1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.155+0000 7f7e02b66700 1 -- 172.21.15.89:0/1001700129 >> 172.21.15.89:0/1001700129 conn(0x7f7dfc0731a0 msgr2=0x7f7dfc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:11.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.155+0000 7f7e02b66700 1 -- 172.21.15.89:0/1001700129 shutdown_connections 2023-11-24T06:21:11.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.155+0000 7f7e02b66700 1 -- 172.21.15.89:0/1001700129 wait complete. 2023-11-24T06:21:11.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.156+0000 7f7e02b66700 1 Processor -- start 2023-11-24T06:21:11.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.156+0000 7f7e02b66700 1 -- start start 2023-11-24T06:21:11.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e02b66700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0b3370 0x7f7dfc157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e02b66700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dfc0d1c70 0x7f7dfc0c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e02b66700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dfc15bdd0 0x7f7dfc0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e02b66700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0c6970 con 0x7f7dfc0b3370 2023-11-24T06:21:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e02b66700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0c5cf0 con 0x7f7dfc0d1c70 2023-11-24T06:21:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e02b66700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0c5fd0 con 0x7f7dfc15bdd0 2023-11-24T06:21:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e00902700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0b3370 0x7f7dfc157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e00902700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0b3370 0x7f7dfc157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39188/0 (socket says 172.21.15.89:39188) 2023-11-24T06:21:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.157+0000 7f7e00902700 1 -- 172.21.15.89:0/4161446938 learned_addr learned my addr 172.21.15.89:0/4161446938 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:11.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e01103700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dfc15bdd0 0x7f7dfc0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7dfbfff700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dfc0d1c70 0x7f7dfc0c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e00902700 1 -- 172.21.15.89:0/4161446938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dfc15bdd0 msgr2=0x7f7dfc0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e00902700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dfc15bdd0 0x7f7dfc0cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e00902700 1 -- 172.21.15.89:0/4161446938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dfc0d1c70 msgr2=0x7f7dfc0c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e00902700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dfc0d1c70 0x7f7dfc0c7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e00902700 1 -- 172.21.15.89:0/4161446938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df4009b30 con 0x7f7dfc0b3370 2023-11-24T06:21:11.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e00902700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0b3370 0x7f7dfc157580 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f7df000e870 tx=0x7f7df000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7df97fa700 1 -- 172.21.15.89:0/4161446938 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0009590 con 0x7f7dfc0b3370 2023-11-24T06:21:11.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7df97fa700 1 -- 172.21.15.89:0/4161446938 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df000ee70 con 0x7f7dfc0b3370 2023-11-24T06:21:11.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dfc0c62b0 con 0x7f7dfc0b3370 2023-11-24T06:21:11.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7df97fa700 1 -- 172.21.15.89:0/4161446938 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df000ca50 con 0x7f7dfc0b3370 2023-11-24T06:21:11.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.158+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dfc0c6d90 con 0x7f7dfc0b3370 2023-11-24T06:21:11.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.159+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dfc067f60 con 0x7f7dfc0b3370 2023-11-24T06:21:11.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.159+0000 7f7df97fa700 1 -- 172.21.15.89:0/4161446938 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f7df000cbb0 con 0x7f7dfc0b3370 2023-11-24T06:21:11.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.160+0000 7f7df97fa700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7de006af60 0x7f7de006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.160+0000 7f7df97fa700 1 -- 172.21.15.89:0/4161446938 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7df0089550 con 0x7f7dfc0b3370 2023-11-24T06:21:11.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.160+0000 7f7dfbfff700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7de006af60 0x7f7de006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.160+0000 7f7dfbfff700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7de006af60 0x7f7de006d420 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc0c5370 tx=0x7f7de8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.164+0000 7f7df97fa700 1 -- 172.21.15.89:0/4161446938 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7df00546a0 con 0x7f7dfc0b3370 2023-11-24T06:21:11.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.382+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dfc0b2b70 con 0x7f7de006af60 2023-11-24T06:21:11.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.384+0000 7f7df97fa700 1 -- 172.21.15.89:0/4161446938 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7de8001480 con 0x7f7de006af60 2023-11-24T06:21:11.386 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-11-24T06:21:11.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-11-24T06:21:11.387 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2023-11-24T06:21:11.387 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-11-24T06:21:11.387 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-11-24T06:21:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7de006af60 msgr2=0x7f7de006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7de006af60 0x7f7de006d420 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc0c5370 tx=0x7f7de8009040 comp rx=0 tx=0).stop 2023-11-24T06:21:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0b3370 msgr2=0x7f7dfc157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0b3370 0x7f7dfc157580 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f7df000e870 tx=0x7f7df000ec30 comp rx=0 tx=0).stop 2023-11-24T06:21:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 shutdown_connections 2023-11-24T06:21:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dfc15bdd0 0x7f7dfc0cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dfc0b3370 0x7f7dfc157580 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7de006af60 0x7f7de006d420 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 --2- 172.21.15.89:0/4161446938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dfc0d1c70 0x7f7dfc0c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 >> 172.21.15.89:0/4161446938 conn(0x7f7dfc0731a0 msgr2=0x7f7dfc0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 shutdown_connections 2023-11-24T06:21:11.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.386+0000 7f7e02b66700 1 -- 172.21.15.89:0/4161446938 wait complete. 2023-11-24T06:21:11.401 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T06:21:11.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.521+0000 7f853e7b7700 1 Processor -- start 2023-11-24T06:21:11.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.521+0000 7f853e7b7700 1 -- start start 2023-11-24T06:21:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.522+0000 7f853e7b7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f853815be70 0x7f85380cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.522+0000 7f853e7b7700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85380b5ee0 con 0x7f85380add40 2023-11-24T06:21:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.522+0000 7f853e7b7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853806ba00 con 0x7f85380746a0 2023-11-24T06:21:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.522+0000 7f853e7b7700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853806bfc0 con 0x7f853815be70 2023-11-24T06:21:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.522+0000 7f8537fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f853815be70 0x7f85380cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.522+0000 7f85377fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f85380add40 0x7f8538073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42700/0 (socket says 172.21.15.89:42700) 2023-11-24T06:21:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.522+0000 7f85377fe700 1 -- 172.21.15.89:0/1078842181 learned_addr learned my addr 172.21.15.89:0/1078842181 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f8537fff700 1 -- 172.21.15.89:0/1078842181 >> v1:172.21.15.123:6789/0 conn(0x7f85380746a0 legacy=0x7f85380b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f8537fff700 1 -- 172.21.15.89:0/1078842181 >> v1:172.21.15.89:6789/0 conn(0x7f85380add40 legacy=0x7f8538073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f8537fff700 1 -- 172.21.15.89:0/1078842181 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853806bce0 con 0x7f853815be70 2023-11-24T06:21:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f8536ffd700 1 -- 172.21.15.89:0/1078842181 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180852365 0 0) 0x7f85380b5ee0 con 0x7f85380add40 2023-11-24T06:21:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f8537fff700 1 --2- 172.21.15.89:0/1078842181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f853815be70 0x7f85380cfdf0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f852c009840 tx=0x7f852c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7187d60a53ccfb3c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f8536ffd700 1 -- 172.21.15.89:0/1078842181 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c011830 con 0x7f853815be70 2023-11-24T06:21:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f8536ffd700 1 -- 172.21.15.89:0/1078842181 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c011990 con 0x7f853815be70 2023-11-24T06:21:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f8536ffd700 1 -- 172.21.15.89:0/1078842181 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c011d20 con 0x7f853815be70 2023-11-24T06:21:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f853e7b7700 1 -- 172.21.15.89:0/1078842181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f853815be70 msgr2=0x7f85380cfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f853e7b7700 1 --2- 172.21.15.89:0/1078842181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f853815be70 0x7f85380cfdf0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f852c009840 tx=0x7f852c009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f853e7b7700 1 -- 172.21.15.89:0/1078842181 shutdown_connections 2023-11-24T06:21:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f853e7b7700 1 --2- 172.21.15.89:0/1078842181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f853815be70 0x7f85380cfdf0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.523+0000 7f853e7b7700 1 -- 172.21.15.89:0/1078842181 >> 172.21.15.89:0/1078842181 conn(0x7f85380d0bf0 msgr2=0x7f85380d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.524+0000 7f853e7b7700 1 -- 172.21.15.89:0/1078842181 shutdown_connections 2023-11-24T06:21:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.524+0000 7f853e7b7700 1 -- 172.21.15.89:0/1078842181 wait complete. 2023-11-24T06:21:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.525+0000 7f853e7b7700 1 Processor -- start 2023-11-24T06:21:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.525+0000 7f853e7b7700 1 -- start start 2023-11-24T06:21:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.525+0000 7f853e7b7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85380746a0 0x7f8538148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f853e7b7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85380add40 0x7f853809e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f853e7b7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815be70 0x7f853809c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f853e7b7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853815ce10 con 0x7f85380746a0 2023-11-24T06:21:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f853e7b7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853809e8a0 con 0x7f853815be70 2023-11-24T06:21:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f853e7b7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853809bf60 con 0x7f85380add40 2023-11-24T06:21:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85380746a0 0x7f8538148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85380746a0 0x7f8538148410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39200/0 (socket says 172.21.15.89:39200) 2023-11-24T06:21:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 -- 172.21.15.89:0/512100607 learned_addr learned my addr 172.21.15.89:0/512100607 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f85377fe700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85380add40 0x7f853809e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f853cd54700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815be70 0x7f853809c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 -- 172.21.15.89:0/512100607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85380add40 msgr2=0x7f853809e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85380add40 0x7f853809e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 -- 172.21.15.89:0/512100607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815be70 msgr2=0x7f853809c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815be70 0x7f853809c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 -- 172.21.15.89:0/512100607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c0094a0 con 0x7f85380746a0 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8537fff700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85380746a0 0x7f8538148410 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f852c014040 tx=0x7f852c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8534ff9700 1 -- 172.21.15.89:0/512100607 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c01a980 con 0x7f85380746a0 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8534ff9700 1 -- 172.21.15.89:0/512100607 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c01ab00 con 0x7f85380746a0 2023-11-24T06:21:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853815c470 con 0x7f85380746a0 2023-11-24T06:21:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f8534ff9700 1 -- 172.21.15.89:0/512100607 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c024420 con 0x7f85380746a0 2023-11-24T06:21:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.526+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853815ca30 con 0x7f85380746a0 2023-11-24T06:21:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.527+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8538068fb0 con 0x7f85380746a0 2023-11-24T06:21:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.528+0000 7f8534ff9700 1 -- 172.21.15.89:0/512100607 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f852c022070 con 0x7f85380746a0 2023-11-24T06:21:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.528+0000 7f8534ff9700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f852406af60 0x7f852406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.528+0000 7f8534ff9700 1 -- 172.21.15.89:0/512100607 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f852c08a880 con 0x7f85380746a0 2023-11-24T06:21:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.528+0000 7f85377fe700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f852406af60 0x7f852406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.528+0000 7f85377fe700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f852406af60 0x7f852406d420 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8528007fd0 tx=0x7f8528007c20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.532+0000 7f8534ff9700 1 -- 172.21.15.89:0/512100607 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f852c055af0 con 0x7f85380746a0 2023-11-24T06:21:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.723+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f85380b39e0 con 0x7f85380746a0 2023-11-24T06:21:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.724+0000 7f8534ff9700 1 -- 172.21.15.89:0/512100607 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f85380b39e0 con 0x7f85380746a0 2023-11-24T06:21:11.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f852406af60 msgr2=0x7f852406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f852406af60 0x7f852406d420 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8528007fd0 tx=0x7f8528007c20 comp rx=0 tx=0).stop 2023-11-24T06:21:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85380746a0 msgr2=0x7f8538148410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85380746a0 0x7f8538148410 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f852c014040 tx=0x7f852c004750 comp rx=0 tx=0).stop 2023-11-24T06:21:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 shutdown_connections 2023-11-24T06:21:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f852406af60 0x7f852406d420 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815be70 0x7f853809c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85380746a0 0x7f8538148410 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 --2- 172.21.15.89:0/512100607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85380add40 0x7f853809e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 >> 172.21.15.89:0/512100607 conn(0x7f85380d0bf0 msgr2=0x7f85380af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 shutdown_connections 2023-11-24T06:21:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.726+0000 7f853e7b7700 1 -- 172.21.15.89:0/512100607 wait complete. 2023-11-24T06:21:11.741 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol1628 --mode=777 2023-11-24T06:21:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.863+0000 7f38de999700 1 Processor -- start 2023-11-24T06:21:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.863+0000 7f38de999700 1 -- start start 2023-11-24T06:21:11.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38de999700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f38d8144600 0x7f38d80b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38de999700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38d806e830 0x7f38d806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38de999700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d806f1c0 con 0x7f38d806f480 2023-11-24T06:21:11.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38de999700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d806aff0 con 0x7f38d806e830 2023-11-24T06:21:11.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38de999700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d806c9b0 con 0x7f38d8144600 2023-11-24T06:21:11.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38dcf36700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38d806e830 0x7f38d806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38dcf36700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f38d806e830 0x7f38d806ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39212/0 (socket says 172.21.15.89:39212) 2023-11-24T06:21:11.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38dcf36700 1 -- 172.21.15.89:0/2410906294 learned_addr learned my addr 172.21.15.89:0/2410906294 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:11.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38d7fff700 1 --2- 172.21.15.89:0/2410906294 >> v2:172.21.15.123:3300/0 conn(0x7f38d8144600 0x7f38d80b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38dcf36700 1 -- 172.21.15.89:0/2410906294 >> v2:172.21.15.123:3300/0 conn(0x7f38d8144600 msgr2=0x7f38d80b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:11.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38dcf36700 1 --2- 172.21.15.89:0/2410906294 >> v2:172.21.15.123:3300/0 conn(0x7f38d8144600 0x7f38d80b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38dcf36700 1 -- 172.21.15.89:0/2410906294 >> v1:172.21.15.123:6789/0 conn(0x7f38d806f480 legacy=0x7f38d806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38dcf36700 1 -- 172.21.15.89:0/2410906294 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d806b200 con 0x7f38d806e830 2023-11-24T06:21:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38dcf36700 1 --2- 172.21.15.89:0/2410906294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d806e830 0x7f38d806ec80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f38d80734a0 tx=0x7f38d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767a8502717c7bff server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.864+0000 7f38d6ffd700 1 -- 172.21.15.89:0/2410906294 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d000e070 con 0x7f38d806e830 2023-11-24T06:21:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38d6ffd700 1 -- 172.21.15.89:0/2410906294 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d000b9c0 con 0x7f38d806e830 2023-11-24T06:21:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38d6ffd700 1 -- 172.21.15.89:0/2410906294 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d000bd70 con 0x7f38d806e830 2023-11-24T06:21:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38de999700 1 -- 172.21.15.89:0/2410906294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d806e830 msgr2=0x7f38d806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38de999700 1 --2- 172.21.15.89:0/2410906294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d806e830 0x7f38d806ec80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f38d80734a0 tx=0x7f38d000b330 comp rx=0 tx=0).stop 2023-11-24T06:21:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38de999700 1 -- 172.21.15.89:0/2410906294 shutdown_connections 2023-11-24T06:21:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38de999700 1 --2- 172.21.15.89:0/2410906294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d806e830 0x7f38d806ec80 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38de999700 1 --2- 172.21.15.89:0/2410906294 >> v2:172.21.15.123:3300/0 conn(0x7f38d8144600 0x7f38d80b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38de999700 1 -- 172.21.15.89:0/2410906294 >> 172.21.15.89:0/2410906294 conn(0x7f38d80d0b40 msgr2=0x7f38d80d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38de999700 1 -- 172.21.15.89:0/2410906294 shutdown_connections 2023-11-24T06:21:11.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.865+0000 7f38de999700 1 -- 172.21.15.89:0/2410906294 wait complete. 2023-11-24T06:21:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.866+0000 7f38de999700 1 Processor -- start 2023-11-24T06:21:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.866+0000 7f38de999700 1 -- start start 2023-11-24T06:21:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38de999700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38d806e830 0x7f38d815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38de999700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38d806f480 0x7f38d814fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38de999700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8144600 0x7f38d813a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38de999700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d815a620 con 0x7f38d8144600 2023-11-24T06:21:11.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38de999700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d81501e0 con 0x7f38d806f480 2023-11-24T06:21:11.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38de999700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d814d580 con 0x7f38d806e830 2023-11-24T06:21:11.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38dcf36700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8144600 0x7f38d813a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38dcf36700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8144600 0x7f38d813a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39218/0 (socket says 172.21.15.89:39218) 2023-11-24T06:21:11.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38dcf36700 1 -- 172.21.15.89:0/3277643241 learned_addr learned my addr 172.21.15.89:0/3277643241 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:11.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38d7fff700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38d806e830 0x7f38d815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.867+0000 7f38d77fe700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38d806f480 0x7f38d814fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38dcf36700 1 -- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38d806e830 msgr2=0x7f38d815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38dcf36700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38d806e830 0x7f38d815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38dcf36700 1 -- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38d806f480 msgr2=0x7f38d814fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:11.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38dcf36700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38d806f480 0x7f38d814fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:11.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38dcf36700 1 -- 172.21.15.89:0/3277643241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d0009b30 con 0x7f38d8144600 2023-11-24T06:21:11.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38d77fe700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38d806f480 0x7f38d814fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:11.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38dcf36700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8144600 0x7f38d813a370 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f38d8072c70 tx=0x7f38d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38d4ff9700 1 -- 172.21.15.89:0/3277643241 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d0004930 con 0x7f38d8144600 2023-11-24T06:21:11.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38d4ff9700 1 -- 172.21.15.89:0/3277643241 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d00099b0 con 0x7f38d8144600 2023-11-24T06:21:11.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d814d860 con 0x7f38d8144600 2023-11-24T06:21:11.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38d4ff9700 1 -- 172.21.15.89:0/3277643241 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d001c680 con 0x7f38d8144600 2023-11-24T06:21:11.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.868+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d814de20 con 0x7f38d8144600 2023-11-24T06:21:11.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.869+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d8067f60 con 0x7f38d8144600 2023-11-24T06:21:11.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.869+0000 7f38d4ff9700 1 -- 172.21.15.89:0/3277643241 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f38d001c7e0 con 0x7f38d8144600 2023-11-24T06:21:11.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.870+0000 7f38d4ff9700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f38c406af60 0x7f38c406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:11.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.870+0000 7f38d4ff9700 1 -- 172.21.15.89:0/3277643241 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38d008b020 con 0x7f38d8144600 2023-11-24T06:21:11.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.870+0000 7f38d7fff700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f38c406af60 0x7f38c406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:11.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.870+0000 7f38d7fff700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f38c406af60 0x7f38c406d420 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f38d8143fb0 tx=0x7f38cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:11.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:11.874+0000 7f38d4ff9700 1 -- 172.21.15.89:0/3277643241 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38d0056060 con 0x7f38d8144600 2023-11-24T06:21:12.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.095+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol1628", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f38d80b4200 con 0x7f38c406af60 2023-11-24T06:21:12.102 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:21:12.100+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:21:12.103 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:21:12.100+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:21:12.103 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:21:12.100+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:21:12.103 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:21:12.100+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:21:12.103 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-11-24T06:21:12.100+0000 7faae8e15700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:21:12.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.130+0000 7f38d4ff9700 1 -- 172.21.15.89:0/3277643241 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f38d80b4200 con 0x7f38c406af60 2023-11-24T06:21:12.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f38c406af60 msgr2=0x7f38c406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f38c406af60 0x7f38c406d420 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f38d8143fb0 tx=0x7f38cc008040 comp rx=0 tx=0).stop 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8144600 msgr2=0x7f38d813a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8144600 0x7f38d813a370 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f38d8072c70 tx=0x7f38d0004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 shutdown_connections 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8144600 0x7f38d813a370 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38d806e830 0x7f38d815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f38c406af60 0x7f38c406d420 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 --2- 172.21.15.89:0/3277643241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38d806f480 0x7f38d814fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 >> 172.21.15.89:0/3277643241 conn(0x7f38d80d0b40 msgr2=0x7f38d80b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:12.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 shutdown_connections 2023-11-24T06:21:12.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.132+0000 7f38de999700 1 -- 172.21.15.89:0/3277643241 wait complete. 2023-11-24T06:21:12.148 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol1628 2023-11-24T06:21:12.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.269+0000 7fb504a38700 1 Processor -- start 2023-11-24T06:21:12.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.269+0000 7fb504a38700 1 -- start start 2023-11-24T06:21:12.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb504a38700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb50015e160 0x7fb5000b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb504a38700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb5000746a0 0x7fb50006d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb504a38700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5000b3c60 con 0x7fb5000afc90 2023-11-24T06:21:12.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb504a38700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5000b4500 con 0x7fb5000746a0 2023-11-24T06:21:12.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb504a38700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5000b58c0 con 0x7fb50015e160 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fed9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb5000746a0 0x7fb50006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fed9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb5000746a0 0x7fb50006d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39242/0 (socket says 172.21.15.89:39242) 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fdd9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb5000afc90 0x7fb500073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42744/0 (socket says 172.21.15.89:42744) 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fed9d700 1 -- 172.21.15.89:0/543636110 learned_addr learned my addr 172.21.15.89:0/543636110 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fe59c700 1 --2- 172.21.15.89:0/543636110 >> v2:172.21.15.123:3300/0 conn(0x7fb50015e160 0x7fb5000b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fed9d700 1 -- 172.21.15.89:0/543636110 >> v2:172.21.15.123:3300/0 conn(0x7fb50015e160 msgr2=0x7fb5000b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fed9d700 1 --2- 172.21.15.89:0/543636110 >> v2:172.21.15.123:3300/0 conn(0x7fb50015e160 0x7fb5000b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fed9d700 1 -- 172.21.15.89:0/543636110 >> v1:172.21.15.89:6789/0 conn(0x7fb5000afc90 legacy=0x7fb500073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fd59a700 1 -- 172.21.15.89:0/543636110 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722378628 0 0) 0x7fb5000b3c60 con 0x7fb5000afc90 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fed9d700 1 -- 172.21.15.89:0/543636110 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5000b0140 con 0x7fb5000746a0 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fed9d700 1 --2- 172.21.15.89:0/543636110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000746a0 0x7fb50006d930 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb50009efa0 tx=0x7fb4f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4c971193745f89 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.270+0000 7fb4fd59a700 1 -- 172.21.15.89:0/543636110 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800e070 con 0x7fb5000746a0 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb4fd59a700 1 -- 172.21.15.89:0/543636110 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f800b9c0 con 0x7fb5000746a0 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb4fd59a700 1 -- 172.21.15.89:0/543636110 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800bd70 con 0x7fb5000746a0 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb504a38700 1 -- 172.21.15.89:0/543636110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000746a0 msgr2=0x7fb50006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb504a38700 1 --2- 172.21.15.89:0/543636110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000746a0 0x7fb50006d930 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb50009efa0 tx=0x7fb4f800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb504a38700 1 -- 172.21.15.89:0/543636110 shutdown_connections 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb504a38700 1 --2- 172.21.15.89:0/543636110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000746a0 0x7fb50006d930 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb504a38700 1 --2- 172.21.15.89:0/543636110 >> v2:172.21.15.123:3300/0 conn(0x7fb50015e160 0x7fb5000b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb504a38700 1 -- 172.21.15.89:0/543636110 >> 172.21.15.89:0/543636110 conn(0x7fb5000d0bf0 msgr2=0x7fb5000d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb504a38700 1 -- 172.21.15.89:0/543636110 shutdown_connections 2023-11-24T06:21:12.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.271+0000 7fb504a38700 1 -- 172.21.15.89:0/543636110 wait complete. 2023-11-24T06:21:12.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.272+0000 7fb504a38700 1 Processor -- start 2023-11-24T06:21:12.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb504a38700 1 -- start start 2023-11-24T06:21:12.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb504a38700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5000746a0 0x7fb5001f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb504a38700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000afc90 0x7fb5001f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb504a38700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb50015e160 0x7fb5001f7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb504a38700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50014b210 con 0x7fb5000afc90 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb504a38700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5001f3990 con 0x7fb5000746a0 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb504a38700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5000c75f0 con 0x7fb50015e160 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb4fdd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000afc90 0x7fb5001f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb4fdd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000afc90 0x7fb5001f7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39246/0 (socket says 172.21.15.89:39246) 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb4fdd9b700 1 -- 172.21.15.89:0/543220230 learned_addr learned my addr 172.21.15.89:0/543220230 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb4fe59c700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5000746a0 0x7fb5001f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.273+0000 7fb4fed9d700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb50015e160 0x7fb5001f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4fdd9b700 1 -- 172.21.15.89:0/543220230 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb50015e160 msgr2=0x7fb5001f7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4fdd9b700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb50015e160 0x7fb5001f7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4fdd9b700 1 -- 172.21.15.89:0/543220230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5000746a0 msgr2=0x7fb5001f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4fdd9b700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5000746a0 0x7fb5001f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4fdd9b700 1 -- 172.21.15.89:0/543220230 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f8009b30 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4fdd9b700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000afc90 0x7fb5001f7110 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40011e0 tx=0x7fb4f400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4eeffd700 1 -- 172.21.15.89:0/543220230 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4004c80 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4eeffd700 1 -- 172.21.15.89:0/543220230 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f4004de0 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50014b540 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb4eeffd700 1 -- 172.21.15.89:0/543220230 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4007310 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.274+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50014bb00 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.275+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50014bc90 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.275+0000 7fb4eeffd700 1 -- 172.21.15.89:0/543220230 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fb4f4007470 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.276+0000 7fb4eeffd700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4e006af60 0x7fb4e006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.276+0000 7fb4eeffd700 1 -- 172.21.15.89:0/543220230 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4f40892e0 con 0x7fb5000afc90 2023-11-24T06:21:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.276+0000 7fb4fe59c700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4e006af60 0x7fb4e006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.276+0000 7fb4fe59c700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4e006af60 0x7fb4e006d420 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0006b20 tx=0x7fb4f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:12.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.280+0000 7fb4eeffd700 1 -- 172.21.15.89:0/543220230 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4f40543a0 con 0x7fb5000afc90 2023-11-24T06:21:12.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.506+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol1628", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5000b4190 con 0x7fb4e006af60 2023-11-24T06:21:12.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.512+0000 7fb4eeffd700 1 -- 172.21.15.89:0/543220230 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fb4f00076b0 con 0x7fb4e006af60 2023-11-24T06:21:12.515 INFO:teuthology.orchestra.run.smithi089.stdout:/volumes/_nogroup/subvol1628/3fffb857-dacd-44b2-830c-ef35d2aeb9de 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4e006af60 msgr2=0x7fb4e006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4e006af60 0x7fb4e006d420 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0006b20 tx=0x7fb4f0008040 comp rx=0 tx=0).stop 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000afc90 msgr2=0x7fb5001f7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000afc90 0x7fb5001f7110 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40011e0 tx=0x7fb4f400ca30 comp rx=0 tx=0).stop 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 shutdown_connections 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb50015e160 0x7fb5001f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb4e006af60 0x7fb4e006d420 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5000746a0 0x7fb5001f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 --2- 172.21.15.89:0/543220230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5000afc90 0x7fb5001f7110 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 >> 172.21.15.89:0/543220230 conn(0x7fb5000d0bf0 msgr2=0x7fb50006e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:12.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 shutdown_connections 2023-11-24T06:21:12.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.514+0000 7fb504a38700 1 -- 172.21.15.89:0/543220230 wait complete. 2023-11-24T06:21:12.529 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol1628 with 3000 files each 0MB size under directory volumes/_nogroup/subvol1628/3fffb857-dacd-44b2-830c-ef35d2aeb9de 2023-11-24T06:21:12.529 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T06:21:12.530 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:21:12.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:21:12.626 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T06:21:12.627 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:21:12.627 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:12.644 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:21:12.644 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:12.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:21:12.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.859+0000 7ff3ac982700 1 Processor -- start 2023-11-24T06:21:12.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.859+0000 7ff3ac982700 1 -- start start 2023-11-24T06:21:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3ac982700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 0x7ff3a81604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3ac982700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff3a80d1c70 0x7ff3a80d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3ac982700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a80b22d0 con 0x7ff3a815bdd0 2023-11-24T06:21:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3ac982700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a80b2f50 con 0x7ff3a80d1c70 2023-11-24T06:21:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3ac982700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a80d2640 con 0x7ff3a80b3370 2023-11-24T06:21:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a6d9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff3a80d1c70 0x7ff3a80d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a659c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff3a815bdd0 0x7ff3a80b1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42756/0 (socket says 172.21.15.89:42756) 2023-11-24T06:21:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a6d9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff3a80d1c70 0x7ff3a80d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39256/0 (socket says 172.21.15.89:39256) 2023-11-24T06:21:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a659c700 1 -- 172.21.15.89:0/493703939 learned_addr learned my addr 172.21.15.89:0/493703939 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a5d9b700 1 --2- 172.21.15.89:0/493703939 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 0x7ff3a81604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a6d9d700 1 -- 172.21.15.89:0/493703939 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 msgr2=0x7ff3a81604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a6d9d700 1 --2- 172.21.15.89:0/493703939 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 0x7ff3a81604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a6d9d700 1 -- 172.21.15.89:0/493703939 >> v1:172.21.15.89:6789/0 conn(0x7ff3a815bdd0 legacy=0x7ff3a80b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a559a700 1 -- 172.21.15.89:0/493703939 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136737735 0 0) 0x7ff3a80b22d0 con 0x7ff3a815bdd0 2023-11-24T06:21:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a6d9d700 1 -- 172.21.15.89:0/493703939 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a80b25b0 con 0x7ff3a80d1c70 2023-11-24T06:21:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a6d9d700 1 --2- 172.21.15.89:0/493703939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a80d1c70 0x7ff3a80d2100 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009d90 tx=0x7ff3a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0dfb8441c0fa50 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.860+0000 7ff3a559a700 1 -- 172.21.15.89:0/493703939 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a000e070 con 0x7ff3a80d1c70 2023-11-24T06:21:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3a559a700 1 -- 172.21.15.89:0/493703939 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a000bbd0 con 0x7ff3a80d1c70 2023-11-24T06:21:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3a559a700 1 -- 172.21.15.89:0/493703939 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0012430 con 0x7ff3a80d1c70 2023-11-24T06:21:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3ac982700 1 -- 172.21.15.89:0/493703939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a80d1c70 msgr2=0x7ff3a80d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3ac982700 1 --2- 172.21.15.89:0/493703939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a80d1c70 0x7ff3a80d2100 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009d90 tx=0x7ff3a000b330 comp rx=0 tx=0).stop 2023-11-24T06:21:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3ac982700 1 -- 172.21.15.89:0/493703939 shutdown_connections 2023-11-24T06:21:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3ac982700 1 --2- 172.21.15.89:0/493703939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a80d1c70 0x7ff3a80d2100 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3ac982700 1 --2- 172.21.15.89:0/493703939 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 0x7ff3a81604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3ac982700 1 -- 172.21.15.89:0/493703939 >> 172.21.15.89:0/493703939 conn(0x7ff3a80731a0 msgr2=0x7ff3a809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3ac982700 1 -- 172.21.15.89:0/493703939 shutdown_connections 2023-11-24T06:21:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.861+0000 7ff3ac982700 1 -- 172.21.15.89:0/493703939 wait complete. 2023-11-24T06:21:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.862+0000 7ff3ac982700 1 Processor -- start 2023-11-24T06:21:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3ac982700 1 -- start start 2023-11-24T06:21:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3ac982700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 0x7ff3a8157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3ac982700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a80d1c70 0x7ff3a81e6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3ac982700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a815bdd0 0x7ff3a81e6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3ac982700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a80cad30 con 0x7ff3a815bdd0 2023-11-24T06:21:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3ac982700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a80c8f50 con 0x7ff3a80d1c70 2023-11-24T06:21:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3ac982700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a81e74e0 con 0x7ff3a80b3370 2023-11-24T06:21:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3a6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a815bdd0 0x7ff3a81e6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3a6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a815bdd0 0x7ff3a81e6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39262/0 (socket says 172.21.15.89:39262) 2023-11-24T06:21:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.863+0000 7ff3a6d9d700 1 -- 172.21.15.89:0/2976071721 learned_addr learned my addr 172.21.15.89:0/2976071721 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3a659c700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 0x7ff3a8157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3a5d9b700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a80d1c70 0x7ff3a81e6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3a6d9d700 1 -- 172.21.15.89:0/2976071721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 msgr2=0x7ff3a8157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3a6d9d700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 0x7ff3a8157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3a6d9d700 1 -- 172.21.15.89:0/2976071721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a80d1c70 msgr2=0x7ff3a81e6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3a6d9d700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a80d1c70 0x7ff3a81e6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3a6d9d700 1 -- 172.21.15.89:0/2976071721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a0009a70 con 0x7ff3a815bdd0 2023-11-24T06:21:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3a6d9d700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a815bdd0 0x7ff3a81e6ea0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80716d0 tx=0x7ff3a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff396ffd700 1 -- 172.21.15.89:0/2976071721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0003ea0 con 0x7ff3a815bdd0 2023-11-24T06:21:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff396ffd700 1 -- 172.21.15.89:0/2976071721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0020070 con 0x7ff3a815bdd0 2023-11-24T06:21:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a80c91f0 con 0x7ff3a815bdd0 2023-11-24T06:21:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff396ffd700 1 -- 172.21.15.89:0/2976071721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0004280 con 0x7ff3a815bdd0 2023-11-24T06:21:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.864+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a80c97b0 con 0x7ff3a815bdd0 2023-11-24T06:21:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.865+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a80c9940 con 0x7ff3a815bdd0 2023-11-24T06:21:12.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.865+0000 7ff396ffd700 1 -- 172.21.15.89:0/2976071721 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ff3a00043e0 con 0x7ff3a815bdd0 2023-11-24T06:21:12.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.866+0000 7ff396ffd700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff38806af60 0x7ff38806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:12.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.866+0000 7ff396ffd700 1 -- 172.21.15.89:0/2976071721 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3a0022070 con 0x7ff3a815bdd0 2023-11-24T06:21:12.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.866+0000 7ff3a659c700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff38806af60 0x7ff38806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:12.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.866+0000 7ff3a659c700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff38806af60 0x7ff38806d420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff398000f60 tx=0x7ff398007c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:12.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:12.870+0000 7ff396ffd700 1 -- 172.21.15.89:0/2976071721 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3a00567d0 con 0x7ff3a815bdd0 2023-11-24T06:21:13.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.032+0000 7ff396ffd700 1 -- 172.21.15.89:0/2976071721 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff3a0056ba0 con 0x7ff3a815bdd0 2023-11-24T06:21:13.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.065+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3a8067f60 con 0x7ff3a815bdd0 2023-11-24T06:21:13.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.066+0000 7ff396ffd700 1 -- 172.21.15.89:0/2976071721 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16532 (secure 0 0 0) 0x7ff3a005db50 con 0x7ff3a815bdd0 2023-11-24T06:21:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:13.070 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":64,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:58.434050+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:55.237439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:55.790741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:21:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff38806af60 msgr2=0x7ff38806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff38806af60 0x7ff38806d420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff398000f60 tx=0x7ff398007c90 comp rx=0 tx=0).stop 2023-11-24T06:21:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a815bdd0 msgr2=0x7ff3a81e6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a815bdd0 0x7ff3a81e6ea0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80716d0 tx=0x7ff3a0004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 shutdown_connections 2023-11-24T06:21:13.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3a815bdd0 0x7ff3a81e6ea0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a80b3370 0x7ff3a8157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff38806af60 0x7ff38806d420 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 --2- 172.21.15.89:0/2976071721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a80d1c70 0x7ff3a81e6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 >> 172.21.15.89:0/2976071721 conn(0x7ff3a80731a0 msgr2=0x7ff3a80aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:13.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 shutdown_connections 2023-11-24T06:21:13.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.069+0000 7ff3ac982700 1 -- 172.21.15.89:0/2976071721 wait complete. 2023-11-24T06:21:13.084 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:20:55.237439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:20:55.790741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:21:13.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:21:13.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.204+0000 7f87b4f6f700 1 Processor -- start 2023-11-24T06:21:13.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.204+0000 7f87b4f6f700 1 -- start start 2023-11-24T06:21:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.205+0000 7f87b4f6f700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f87b00cf110 0x7f87b00cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.205+0000 7f87b4f6f700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b00d0140 con 0x7f87b00c6c30 2023-11-24T06:21:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.205+0000 7f87b4f6f700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b00d03e0 con 0x7f87b015bf50 2023-11-24T06:21:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.205+0000 7f87b4f6f700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b015c330 con 0x7f87b00cf110 2023-11-24T06:21:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.205+0000 7f87add9b700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f87b00c6c30 0x7f87b00cea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42772/0 (socket says 172.21.15.89:42772) 2023-11-24T06:21:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.205+0000 7f87aed9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f87b00cf110 0x7f87b00cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.205+0000 7f87add9b700 1 -- 172.21.15.89:0/3493575413 learned_addr learned my addr 172.21.15.89:0/3493575413 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.206+0000 7f87aed9d700 1 -- 172.21.15.89:0/3493575413 >> v1:172.21.15.123:6789/0 conn(0x7f87b015bf50 legacy=0x7f87b00ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.206+0000 7f87aed9d700 1 -- 172.21.15.89:0/3493575413 >> v1:172.21.15.89:6789/0 conn(0x7f87b00c6c30 legacy=0x7f87b00cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.206+0000 7f87aed9d700 1 -- 172.21.15.89:0/3493575413 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b00d06c0 con 0x7f87b00cf110 2023-11-24T06:21:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.206+0000 7f87ad59a700 1 -- 172.21.15.89:0/3493575413 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847287550 0 0) 0x7f87b00d0140 con 0x7f87b00c6c30 2023-11-24T06:21:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.206+0000 7f87aed9d700 1 --2- 172.21.15.89:0/3493575413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b00cf110 0x7f87b00cd6d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009d90 tx=0x7f87a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c090062cdccde45f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.206+0000 7f87ad59a700 1 -- 172.21.15.89:0/3493575413 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800e070 con 0x7f87b00cf110 2023-11-24T06:21:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.206+0000 7f87ad59a700 1 -- 172.21.15.89:0/3493575413 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a800bb90 con 0x7f87b00cf110 2023-11-24T06:21:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.206+0000 7f87ad59a700 1 -- 172.21.15.89:0/3493575413 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8003a20 con 0x7f87b00cf110 2023-11-24T06:21:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.207+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3493575413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b00cf110 msgr2=0x7f87b00cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.207+0000 7f87b4f6f700 1 --2- 172.21.15.89:0/3493575413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b00cf110 0x7f87b00cd6d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009d90 tx=0x7f87a800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.207+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3493575413 shutdown_connections 2023-11-24T06:21:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.207+0000 7f87b4f6f700 1 --2- 172.21.15.89:0/3493575413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b00cf110 0x7f87b00cd6d0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.207+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3493575413 >> 172.21.15.89:0/3493575413 conn(0x7f87b00b3780 msgr2=0x7f87b00b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.207+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3493575413 shutdown_connections 2023-11-24T06:21:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.207+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3493575413 wait complete. 2023-11-24T06:21:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.208+0000 7f87b4f6f700 1 Processor -- start 2023-11-24T06:21:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.208+0000 7f87b4f6f700 1 -- start start 2023-11-24T06:21:13.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87b4f6f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87b00c6c30 0x7f87b015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87b4f6f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b00cf110 0x7f87b00c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87b4f6f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b015bf50 0x7f87b01470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87b4f6f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00cbe50 con 0x7f87b015bf50 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87b4f6f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00cb2d0 con 0x7f87b00cf110 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87b4f6f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00cb5b0 con 0x7f87b00c6c30 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87aed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b015bf50 0x7f87b01470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87aed9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b015bf50 0x7f87b01470d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39276/0 (socket says 172.21.15.89:39276) 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.209+0000 7f87aed9d700 1 -- 172.21.15.89:0/3898285459 learned_addr learned my addr 172.21.15.89:0/3898285459 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87add9b700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b00cf110 0x7f87b00c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87ae59c700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87b00c6c30 0x7f87b015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87aed9d700 1 -- 172.21.15.89:0/3898285459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87b00c6c30 msgr2=0x7f87b015b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87aed9d700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87b00c6c30 0x7f87b015b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87aed9d700 1 -- 172.21.15.89:0/3898285459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b00cf110 msgr2=0x7f87b00c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87aed9d700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b00cf110 0x7f87b00c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87aed9d700 1 -- 172.21.15.89:0/3898285459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a8009a70 con 0x7f87b015bf50 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87aed9d700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b015bf50 0x7f87b01470d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f87b00723f0 tx=0x7f87a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f879effd700 1 -- 172.21.15.89:0/3898285459 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8012630 con 0x7f87b015bf50 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f879effd700 1 -- 172.21.15.89:0/3898285459 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a8012790 con 0x7f87b015bf50 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b00cb890 con 0x7f87b015bf50 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f879effd700 1 -- 172.21.15.89:0/3898285459 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8012b40 con 0x7f87b015bf50 2023-11-24T06:21:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.210+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b0147610 con 0x7f87b015bf50 2023-11-24T06:21:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.211+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b0068fb0 con 0x7f87b015bf50 2023-11-24T06:21:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.211+0000 7f879effd700 1 -- 172.21.15.89:0/3898285459 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f87a8012ca0 con 0x7f87b015bf50 2023-11-24T06:21:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.212+0000 7f879effd700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f879006aff0 0x7f879006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.212+0000 7f879effd700 1 -- 172.21.15.89:0/3898285459 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87a808bac0 con 0x7f87b015bf50 2023-11-24T06:21:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.212+0000 7f87ae59c700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f879006aff0 0x7f879006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.212+0000 7f87ae59c700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f879006aff0 0x7f879006d4b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f87a4009870 tx=0x7f87a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.216+0000 7f879effd700 1 -- 172.21.15.89:0/3898285459 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87a80569e0 con 0x7f87b015bf50 2023-11-24T06:21:13.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.405+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87b0147870 con 0x7f87b015bf50 2023-11-24T06:21:13.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.405+0000 7f879effd700 1 -- 172.21.15.89:0/3898285459 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87a805a000 con 0x7f87b015bf50 2023-11-24T06:21:13.409 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:21:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f879006aff0 msgr2=0x7f879006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f879006aff0 0x7f879006d4b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f87a4009870 tx=0x7f87a4008040 comp rx=0 tx=0).stop 2023-11-24T06:21:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b015bf50 msgr2=0x7f87b01470d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b015bf50 0x7f87b01470d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f87b00723f0 tx=0x7f87a8004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 shutdown_connections 2023-11-24T06:21:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87b015bf50 0x7f87b01470d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87b00c6c30 0x7f87b015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f879006aff0 0x7f879006d4b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 --2- 172.21.15.89:0/3898285459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87b00cf110 0x7f87b00c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 >> 172.21.15.89:0/3898285459 conn(0x7f87b00b3780 msgr2=0x7f87b00c70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 shutdown_connections 2023-11-24T06:21:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.407+0000 7f87b4f6f700 1 -- 172.21.15.89:0/3898285459 wait complete. 2023-11-24T06:21:13.423 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:21:13.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.545+0000 7f801cb8f700 1 Processor -- start 2023-11-24T06:21:13.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.545+0000 7f801cb8f700 1 -- start start 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f801cb8f700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f801815e160 0x7f80180b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f801cb8f700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f80180afc90 0x7f8018073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f801cb8f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180744d0 0x7f801806d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f801cb8f700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80180b4500 con 0x7f80180afc90 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f801cb8f700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8018074960 con 0x7f801815e160 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f801cb8f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80180b3c60 con 0x7f80180744d0 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8015d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f80180afc90 0x7f8018073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8015d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f80180afc90 0x7f8018073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39280/0 (socket says 172.21.15.89:39280) 2023-11-24T06:21:13.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8015d9b700 1 -- 172.21.15.89:0/2691340647 learned_addr learned my addr 172.21.15.89:0/2691340647 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8016d9d700 1 --2- 172.21.15.89:0/2691340647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180744d0 0x7f801806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f801659c700 1 --2- 172.21.15.89:0/2691340647 >> v2:172.21.15.123:3300/0 conn(0x7f801815e160 0x7f80180b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8015d9b700 1 -- 172.21.15.89:0/2691340647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180744d0 msgr2=0x7f801806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8015d9b700 1 --2- 172.21.15.89:0/2691340647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180744d0 0x7f801806d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8015d9b700 1 -- 172.21.15.89:0/2691340647 >> v2:172.21.15.123:3300/0 conn(0x7f801815e160 msgr2=0x7f80180b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8015d9b700 1 --2- 172.21.15.89:0/2691340647 >> v2:172.21.15.123:3300/0 conn(0x7f801815e160 0x7f80180b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f8015d9b700 1 -- 172.21.15.89:0/2691340647 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80180b58c0 con 0x7f80180afc90 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.546+0000 7f801659c700 1 --2- 172.21.15.89:0/2691340647 >> v2:172.21.15.123:3300/0 conn(0x7f801815e160 0x7f80180b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f8015d9b700 1 --2- 172.21.15.89:0/2691340647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80180afc90 0x7f8018073f90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f801809efa0 tx=0x7f800c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943a5d66b6ad185d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801559a700 1 -- 172.21.15.89:0/2691340647 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c00b870 con 0x7f80180afc90 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801559a700 1 -- 172.21.15.89:0/2691340647 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800c00b9d0 con 0x7f80180afc90 2023-11-24T06:21:13.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801559a700 1 -- 172.21.15.89:0/2691340647 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c00bd60 con 0x7f80180afc90 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 -- 172.21.15.89:0/2691340647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80180afc90 msgr2=0x7f8018073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 --2- 172.21.15.89:0/2691340647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80180afc90 0x7f8018073f90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f801809efa0 tx=0x7f800c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 -- 172.21.15.89:0/2691340647 shutdown_connections 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 --2- 172.21.15.89:0/2691340647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180744d0 0x7f801806d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 --2- 172.21.15.89:0/2691340647 >> v2:172.21.15.123:3300/0 conn(0x7f801815e160 0x7f80180b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 --2- 172.21.15.89:0/2691340647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80180afc90 0x7f8018073f90 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 -- 172.21.15.89:0/2691340647 >> 172.21.15.89:0/2691340647 conn(0x7f80180d0bf0 msgr2=0x7f80180d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 -- 172.21.15.89:0/2691340647 shutdown_connections 2023-11-24T06:21:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.547+0000 7f801cb8f700 1 -- 172.21.15.89:0/2691340647 wait complete. 2023-11-24T06:21:13.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.549+0000 7f801cb8f700 1 Processor -- start 2023-11-24T06:21:13.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.549+0000 7f801cb8f700 1 -- start start 2023-11-24T06:21:13.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.549+0000 7f801cb8f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80180744d0 0x7f80181525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.549+0000 7f801cb8f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180afc90 0x7f801815f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.549+0000 7f801cb8f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f801815e160 0x7f8018156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.549+0000 7f801cb8f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801815e540 con 0x7f801815e160 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.549+0000 7f801cb8f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801815e790 con 0x7f80180744d0 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.549+0000 7f801cb8f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801815ea70 con 0x7f80180afc90 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f801815e160 0x7f8018156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f801815e160 0x7f8018156810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39288/0 (socket says 172.21.15.89:39288) 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 -- 172.21.15.89:0/3057559157 learned_addr learned my addr 172.21.15.89:0/3057559157 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f801659c700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80180744d0 0x7f80181525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8015d9b700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180afc90 0x7f801815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 -- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180afc90 msgr2=0x7f801815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180afc90 0x7f801815f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 -- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80180744d0 msgr2=0x7f80181525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80180744d0 0x7f80181525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 -- 172.21.15.89:0/3057559157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c009550 con 0x7f801815e160 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f801659c700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80180744d0 0x7f80181525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8016d9d700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f801815e160 0x7f8018156810 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f801000e810 tx=0x7f801000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8006ffd700 1 -- 172.21.15.89:0/3057559157 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010013610 con 0x7f801815e160 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8006ffd700 1 -- 172.21.15.89:0/3057559157 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010013770 con 0x7f801815e160 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801815ed50 con 0x7f801815e160 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f8006ffd700 1 -- 172.21.15.89:0/3057559157 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010013b20 con 0x7f801815e160 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.550+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801815f310 con 0x7f801815e160 2023-11-24T06:21:13.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.551+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801815f4a0 con 0x7f801815e160 2023-11-24T06:21:13.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.552+0000 7f8006ffd700 1 -- 172.21.15.89:0/3057559157 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8010013c80 con 0x7f801815e160 2023-11-24T06:21:13.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.552+0000 7f8006ffd700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7ff806aff0 0x7f7ff806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.552+0000 7f801659c700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7ff806aff0 0x7f7ff806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.552+0000 7f8006ffd700 1 -- 172.21.15.89:0/3057559157 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8010089540 con 0x7f801815e160 2023-11-24T06:21:13.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.553+0000 7f801659c700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7ff806aff0 0x7f7ff806d4b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8008007ec0 tx=0x7f8008007410 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.557+0000 7f8006ffd700 1 -- 172.21.15.89:0/3057559157 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80100543d0 con 0x7f801815e160 2023-11-24T06:21:13.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.746+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8018068fb0 con 0x7f801815e160 2023-11-24T06:21:13.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.746+0000 7f8006ffd700 1 -- 172.21.15.89:0/3057559157 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v64) v1 ==== 104+0+11 (secure 0 0 0) 0x7f80100579f0 con 0x7f801815e160 2023-11-24T06:21:13.749 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7ff806aff0 msgr2=0x7f7ff806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7ff806aff0 0x7f7ff806d4b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8008007ec0 tx=0x7f8008007410 comp rx=0 tx=0).stop 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f801815e160 msgr2=0x7f8018156810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f801815e160 0x7f8018156810 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f801000e810 tx=0x7f801000ebd0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 shutdown_connections 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f801815e160 0x7f8018156810 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7ff806aff0 0x7f7ff806d4b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80180744d0 0x7f80181525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 --2- 172.21.15.89:0/3057559157 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80180afc90 0x7f801815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.748+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 >> 172.21.15.89:0/3057559157 conn(0x7f80180d0bf0 msgr2=0x7f801806ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.749+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 shutdown_connections 2023-11-24T06:21:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.749+0000 7f801cb8f700 1 -- 172.21.15.89:0/3057559157 wait complete. 2023-11-24T06:21:13.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:21:13.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.883+0000 7f7d8cb38700 1 Processor -- start 2023-11-24T06:21:13.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.883+0000 7f7d8cb38700 1 -- start start 2023-11-24T06:21:13.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.884+0000 7f7d8cb38700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7d880cc540 0x7f7d8806e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.884+0000 7f7d8cb38700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8806ef50 con 0x7f7d8806eb20 2023-11-24T06:21:13.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.884+0000 7f7d8cb38700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880b2f20 con 0x7f7d8815a620 2023-11-24T06:21:13.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.884+0000 7f7d8cb38700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8815aa00 con 0x7f7d880cc540 2023-11-24T06:21:13.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.884+0000 7f7d85d9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7d880cc540 0x7f7d8806e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.884+0000 7f7d86d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7d8806eb20 0x7f7d880b3350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42798/0 (socket says 172.21.15.89:42798) 2023-11-24T06:21:13.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.884+0000 7f7d86d9d700 1 -- 172.21.15.89:0/2387844219 learned_addr learned my addr 172.21.15.89:0/2387844219 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:13.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.885+0000 7f7d85d9b700 1 -- 172.21.15.89:0/2387844219 >> v1:172.21.15.123:6789/0 conn(0x7f7d8815a620 legacy=0x7f7d880ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:13.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.885+0000 7f7d85d9b700 1 -- 172.21.15.89:0/2387844219 >> v1:172.21.15.89:6789/0 conn(0x7f7d8806eb20 legacy=0x7f7d880b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.885+0000 7f7d85d9b700 1 -- 172.21.15.89:0/2387844219 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880ce960 con 0x7f7d880cc540 2023-11-24T06:21:13.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.885+0000 7f7d8559a700 1 -- 172.21.15.89:0/2387844219 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307633445 0 0) 0x7f7d8806ef50 con 0x7f7d8806eb20 2023-11-24T06:21:13.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.885+0000 7f7d85d9b700 1 --2- 172.21.15.89:0/2387844219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d880cc540 0x7f7d8806e5e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009840 tx=0x7f7d78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6796db942421a6d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.885+0000 7f7d8559a700 1 -- 172.21.15.89:0/2387844219 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78004230 con 0x7f7d880cc540 2023-11-24T06:21:13.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.885+0000 7f7d8559a700 1 -- 172.21.15.89:0/2387844219 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d78004390 con 0x7f7d880cc540 2023-11-24T06:21:13.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.885+0000 7f7d8559a700 1 -- 172.21.15.89:0/2387844219 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d780036a0 con 0x7f7d880cc540 2023-11-24T06:21:13.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.886+0000 7f7d8cb38700 1 -- 172.21.15.89:0/2387844219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d880cc540 msgr2=0x7f7d8806e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.886+0000 7f7d8cb38700 1 --2- 172.21.15.89:0/2387844219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d880cc540 0x7f7d8806e5e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009840 tx=0x7f7d78009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:13.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.886+0000 7f7d8cb38700 1 -- 172.21.15.89:0/2387844219 shutdown_connections 2023-11-24T06:21:13.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.886+0000 7f7d8cb38700 1 --2- 172.21.15.89:0/2387844219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d880cc540 0x7f7d8806e5e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.886+0000 7f7d8cb38700 1 -- 172.21.15.89:0/2387844219 >> 172.21.15.89:0/2387844219 conn(0x7f7d880b2050 msgr2=0x7f7d880b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:13.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.886+0000 7f7d8cb38700 1 -- 172.21.15.89:0/2387844219 shutdown_connections 2023-11-24T06:21:13.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.886+0000 7f7d8cb38700 1 -- 172.21.15.89:0/2387844219 wait complete. 2023-11-24T06:21:13.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.887+0000 7f7d8cb38700 1 Processor -- start 2023-11-24T06:21:13.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.887+0000 7f7d8cb38700 1 -- start start 2023-11-24T06:21:13.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8cb38700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d8806eb20 0x7f7d8814bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8cb38700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d880cc540 0x7f7d880ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8cb38700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8815a620 0x7f7d8815ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8cb38700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880abf80 con 0x7f7d8806eb20 2023-11-24T06:21:13.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8cb38700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880ab670 con 0x7f7d8815a620 2023-11-24T06:21:13.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8cb38700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8815d890 con 0x7f7d880cc540 2023-11-24T06:21:13.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d8806eb20 0x7f7d8814bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d8806eb20 0x7f7d8814bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39298/0 (socket says 172.21.15.89:39298) 2023-11-24T06:21:13.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 -- 172.21.15.89:0/1917679844 learned_addr learned my addr 172.21.15.89:0/1917679844 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d86d9d700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8815a620 0x7f7d8815ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d85d9b700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d880cc540 0x7f7d880ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 -- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d880cc540 msgr2=0x7f7d880ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d880cc540 0x7f7d880ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 -- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8815a620 msgr2=0x7f7d8815ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8815a620 0x7f7d8815ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 -- 172.21.15.89:0/1917679844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d780094a0 con 0x7f7d8806eb20 2023-11-24T06:21:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d86d9d700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8815a620 0x7f7d8815ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8659c700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d8806eb20 0x7f7d8814bcb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c00c4d0 tx=0x7f7d7c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d76ffd700 1 -- 172.21.15.89:0/1917679844 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c011790 con 0x7f7d8806eb20 2023-11-24T06:21:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d76ffd700 1 -- 172.21.15.89:0/1917679844 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c0118f0 con 0x7f7d8806eb20 2023-11-24T06:21:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.888+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d880ab8c0 con 0x7f7d8806eb20 2023-11-24T06:21:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.889+0000 7f7d76ffd700 1 -- 172.21.15.89:0/1917679844 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c011ca0 con 0x7f7d8806eb20 2023-11-24T06:21:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.889+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d880abc50 con 0x7f7d8806eb20 2023-11-24T06:21:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.889+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d88068d90 con 0x7f7d8806eb20 2023-11-24T06:21:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.890+0000 7f7d76ffd700 1 -- 172.21.15.89:0/1917679844 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7d7c01c460 con 0x7f7d8806eb20 2023-11-24T06:21:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.890+0000 7f7d76ffd700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7d6806aff0 0x7f7d6806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.890+0000 7f7d76ffd700 1 -- 172.21.15.89:0/1917679844 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d7c089530 con 0x7f7d8806eb20 2023-11-24T06:21:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.890+0000 7f7d85d9b700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7d6806aff0 0x7f7d6806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:13.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.891+0000 7f7d85d9b700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7d6806aff0 0x7f7d6806d4b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000c00 tx=0x7f7d7801c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:13.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:13.895+0000 7f7d76ffd700 1 -- 172.21.15.89:0/1917679844 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d7c054560 con 0x7f7d8806eb20 2023-11-24T06:21:14.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.086+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d88068fb0 con 0x7f7d8806eb20 2023-11-24T06:21:14.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.086+0000 7f7d76ffd700 1 -- 172.21.15.89:0/1917679844 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v64) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d7c057b80 con 0x7f7d8806eb20 2023-11-24T06:21:14.090 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:21:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7d6806aff0 msgr2=0x7f7d6806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7d6806aff0 0x7f7d6806d4b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000c00 tx=0x7f7d7801c040 comp rx=0 tx=0).stop 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d8806eb20 msgr2=0x7f7d8814bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d8806eb20 0x7f7d8814bcb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c00c4d0 tx=0x7f7d7c00c890 comp rx=0 tx=0).stop 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 shutdown_connections 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7d6806aff0 0x7f7d6806d4b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8815a620 0x7f7d8815ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d8806eb20 0x7f7d8814bcb0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 --2- 172.21.15.89:0/1917679844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d880cc540 0x7f7d880ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 >> 172.21.15.89:0/1917679844 conn(0x7f7d880b2050 msgr2=0x7f7d880aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.088+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 shutdown_connections 2023-11-24T06:21:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.089+0000 7f7d8cb38700 1 -- 172.21.15.89:0/1917679844 wait complete. 2023-11-24T06:21:14.103 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:21:14.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.222+0000 7fe999553700 1 Processor -- start 2023-11-24T06:21:14.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.222+0000 7fe999553700 1 -- start start 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe999553700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe99415e1e0 0x7fe9940d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe999553700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe9940d0f80 0x7fe9940b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe999553700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9940d20b0 con 0x7fe9940d3a00 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe999553700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9940d3150 con 0x7fe99415e1e0 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe999553700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe99415e5c0 con 0x7fe9940d0f80 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe992ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe99415e1e0 0x7fe9940d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe992ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe99415e1e0 0x7fe9940d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39306/0 (socket says 172.21.15.89:39306) 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe992ffd700 1 -- 172.21.15.89:0/2061264862 learned_addr learned my addr 172.21.15.89:0/2061264862 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe9937fe700 1 --2- 172.21.15.89:0/2061264862 >> v2:172.21.15.123:3300/0 conn(0x7fe9940d0f80 0x7fe9940b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe992ffd700 1 -- 172.21.15.89:0/2061264862 >> v2:172.21.15.123:3300/0 conn(0x7fe9940d0f80 msgr2=0x7fe9940b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe992ffd700 1 --2- 172.21.15.89:0/2061264862 >> v2:172.21.15.123:3300/0 conn(0x7fe9940d0f80 0x7fe9940b6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe992ffd700 1 -- 172.21.15.89:0/2061264862 >> v1:172.21.15.123:6789/0 conn(0x7fe9940d3a00 legacy=0x7fe9940d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe992ffd700 1 -- 172.21.15.89:0/2061264862 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9940d2390 con 0x7fe99415e1e0 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe9937fe700 1 --2- 172.21.15.89:0/2061264862 >> v2:172.21.15.123:3300/0 conn(0x7fe9940d0f80 0x7fe9940b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe992ffd700 1 --2- 172.21.15.89:0/2061264862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99415e1e0 0x7fe9940d34c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe99408c640 tx=0x7fe988009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceef955beab2a559 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe991ffb700 1 -- 172.21.15.89:0/2061264862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988004230 con 0x7fe99415e1e0 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe991ffb700 1 -- 172.21.15.89:0/2061264862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe988004390 con 0x7fe99415e1e0 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.223+0000 7fe991ffb700 1 -- 172.21.15.89:0/2061264862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9880036a0 con 0x7fe99415e1e0 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.224+0000 7fe999553700 1 -- 172.21.15.89:0/2061264862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99415e1e0 msgr2=0x7fe9940d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.224+0000 7fe999553700 1 --2- 172.21.15.89:0/2061264862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99415e1e0 0x7fe9940d34c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe99408c640 tx=0x7fe988009c00 comp rx=0 tx=0).stop 2023-11-24T06:21:14.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.224+0000 7fe999553700 1 -- 172.21.15.89:0/2061264862 shutdown_connections 2023-11-24T06:21:14.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.224+0000 7fe999553700 1 --2- 172.21.15.89:0/2061264862 >> v2:172.21.15.123:3300/0 conn(0x7fe9940d0f80 0x7fe9940b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.224+0000 7fe999553700 1 --2- 172.21.15.89:0/2061264862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99415e1e0 0x7fe9940d34c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.224+0000 7fe999553700 1 -- 172.21.15.89:0/2061264862 >> 172.21.15.89:0/2061264862 conn(0x7fe9940b5b50 msgr2=0x7fe9940b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:14.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.224+0000 7fe999553700 1 -- 172.21.15.89:0/2061264862 shutdown_connections 2023-11-24T06:21:14.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.224+0000 7fe999553700 1 -- 172.21.15.89:0/2061264862 wait complete. 2023-11-24T06:21:14.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.225+0000 7fe999553700 1 Processor -- start 2023-11-24T06:21:14.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.225+0000 7fe999553700 1 -- start start 2023-11-24T06:21:14.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe999553700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9940d0f80 0x7fe9940ab690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe999553700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9940d3a00 0x7fe9940ac800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe999553700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe99415e1e0 0x7fe9940acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe999553700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9940abf30 con 0x7fe9940d0f80 2023-11-24T06:21:14.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe999553700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9940caf30 con 0x7fe99415e1e0 2023-11-24T06:21:14.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe999553700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9940cb210 con 0x7fe9940d3a00 2023-11-24T06:21:14.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe992ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9940d0f80 0x7fe9940ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe992ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9940d0f80 0x7fe9940ab690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39312/0 (socket says 172.21.15.89:39312) 2023-11-24T06:21:14.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe992ffd700 1 -- 172.21.15.89:0/3537289472 learned_addr learned my addr 172.21.15.89:0/3537289472 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:14.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe992ffd700 1 -- 172.21.15.89:0/3537289472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9940d3a00 msgr2=0x7fe9940ac800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe992ffd700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9940d3a00 0x7fe9940ac800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe992ffd700 1 -- 172.21.15.89:0/3537289472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe99415e1e0 msgr2=0x7fe9940acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe992ffd700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe99415e1e0 0x7fe9940acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.226+0000 7fe992ffd700 1 -- 172.21.15.89:0/3537289472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe988009550 con 0x7fe9940d0f80 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.227+0000 7fe992ffd700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9940d0f80 0x7fe9940ab690 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe988009130 tx=0x7fe988004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.227+0000 7fe97b7fe700 1 -- 172.21.15.89:0/3537289472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988011830 con 0x7fe9940d0f80 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.227+0000 7fe97b7fe700 1 -- 172.21.15.89:0/3537289472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe988011990 con 0x7fe9940d0f80 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.227+0000 7fe97b7fe700 1 -- 172.21.15.89:0/3537289472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988011d00 con 0x7fe9940d0f80 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.227+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9940cb4f0 con 0x7fe9940d0f80 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.227+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9940abbd0 con 0x7fe9940d0f80 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.228+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9940751e0 con 0x7fe9940d0f80 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.228+0000 7fe97b7fe700 1 -- 172.21.15.89:0/3537289472 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe98801b440 con 0x7fe9940d0f80 2023-11-24T06:21:14.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.229+0000 7fe97b7fe700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98006aff0 0x7fe98006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.229+0000 7fe97b7fe700 1 -- 172.21.15.89:0/3537289472 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe98808b050 con 0x7fe9940d0f80 2023-11-24T06:21:14.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.229+0000 7fe9927fc700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98006aff0 0x7fe98006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.229+0000 7fe9927fc700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98006aff0 0x7fe98006d4b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe9940caee0 tx=0x7fe984007a40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:14.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.233+0000 7fe97b7fe700 1 -- 172.21.15.89:0/3537289472 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe988056080 con 0x7fe9940d0f80 2023-11-24T06:21:14.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.439+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe994160a70 con 0x7fe9940d0f80 2023-11-24T06:21:14.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.439+0000 7fe97b7fe700 1 -- 172.21.15.89:0/3537289472 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v64) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe988011e60 con 0x7fe9940d0f80 2023-11-24T06:21:14.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98006aff0 msgr2=0x7fe98006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98006aff0 0x7fe98006d4b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe9940caee0 tx=0x7fe984007a40 comp rx=0 tx=0).stop 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9940d0f80 msgr2=0x7fe9940ab690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9940d0f80 0x7fe9940ab690 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe988009130 tx=0x7fe988004750 comp rx=0 tx=0).stop 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 shutdown_connections 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe98006aff0 0x7fe98006d4b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe99415e1e0 0x7fe9940acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9940d0f80 0x7fe9940ab690 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 --2- 172.21.15.89:0/3537289472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9940d3a00 0x7fe9940ac800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 >> 172.21.15.89:0/3537289472 conn(0x7fe9940b5b50 msgr2=0x7fe9940b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 shutdown_connections 2023-11-24T06:21:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.441+0000 7fe999553700 1 -- 172.21.15.89:0/3537289472 wait complete. 2023-11-24T06:21:14.446 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2023-11-24T06:21:14.456 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T06:21:14.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:14.456 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T06:21:14.501 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:21:14.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:14.501 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T06:21:14.518 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:21:14.518 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:14.540 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:21:14.540 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:14.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:21:14.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.718+0000 7fc6b33b7700 1 Processor -- start 2023-11-24T06:21:14.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.718+0000 7fc6b33b7700 1 -- start start 2023-11-24T06:21:14.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b33b7700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc6ac15bdd0 0x7fc6ac1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b33b7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc6ac0d1c70 0x7fc6ac0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b33b7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ac0b22d0 con 0x7fc6ac0b3370 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b33b7700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac0b2f50 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b33b7700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac0d2640 con 0x7fc6ac15bdd0 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b1954700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc6ac0d1c70 0x7fc6ac0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b1954700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc6ac0d1c70 0x7fc6ac0d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39324/0 (socket says 172.21.15.89:39324) 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b1954700 1 -- 172.21.15.89:0/2825498194 learned_addr learned my addr 172.21.15.89:0/2825498194 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b1153700 1 --2- 172.21.15.89:0/2825498194 >> v2:172.21.15.123:3300/0 conn(0x7fc6ac15bdd0 0x7fc6ac1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b1954700 1 -- 172.21.15.89:0/2825498194 >> v2:172.21.15.123:3300/0 conn(0x7fc6ac15bdd0 msgr2=0x7fc6ac1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b1954700 1 --2- 172.21.15.89:0/2825498194 >> v2:172.21.15.123:3300/0 conn(0x7fc6ac15bdd0 0x7fc6ac1604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b1954700 1 -- 172.21.15.89:0/2825498194 >> v1:172.21.15.123:6789/0 conn(0x7fc6ac0b3370 legacy=0x7fc6ac0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.719+0000 7fc6b1954700 1 -- 172.21.15.89:0/2825498194 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac0b25b0 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b1954700 1 --2- 172.21.15.89:0/2825498194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac0d2100 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac06ec10 tx=0x7fc6a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d2d1537c108f66 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6a3fff700 1 -- 172.21.15.89:0/2825498194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400e070 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6a3fff700 1 -- 172.21.15.89:0/2825498194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a400b9c0 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6a3fff700 1 -- 172.21.15.89:0/2825498194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400bd70 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b33b7700 1 -- 172.21.15.89:0/2825498194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 msgr2=0x7fc6ac0d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/2825498194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac0d2100 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac06ec10 tx=0x7fc6a400b330 comp rx=0 tx=0).stop 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b33b7700 1 -- 172.21.15.89:0/2825498194 shutdown_connections 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/2825498194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac0d2100 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/2825498194 >> v2:172.21.15.123:3300/0 conn(0x7fc6ac15bdd0 0x7fc6ac1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b33b7700 1 -- 172.21.15.89:0/2825498194 >> 172.21.15.89:0/2825498194 conn(0x7fc6ac0731a0 msgr2=0x7fc6ac09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:14.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b33b7700 1 -- 172.21.15.89:0/2825498194 shutdown_connections 2023-11-24T06:21:14.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.720+0000 7fc6b33b7700 1 -- 172.21.15.89:0/2825498194 wait complete. 2023-11-24T06:21:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.721+0000 7fc6b33b7700 1 Processor -- start 2023-11-24T06:21:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b33b7700 1 -- start start 2023-11-24T06:21:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b33b7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6ac0b3370 0x7fc6ac1fba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b33b7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac15c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b33b7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6ac15bdd0 0x7fc6ac1f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b33b7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1fc940 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b33b7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac15e210 con 0x7fc6ac15bdd0 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b33b7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1fc0a0 con 0x7fc6ac0b3370 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b0952700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac15c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b0952700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac15c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39328/0 (socket says 172.21.15.89:39328) 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b0952700 1 -- 172.21.15.89:0/3661937590 learned_addr learned my addr 172.21.15.89:0/3661937590 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.722+0000 7fc6b1954700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6ac15bdd0 0x7fc6ac1f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b1153700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6ac0b3370 0x7fc6ac1fba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b0952700 1 -- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6ac0b3370 msgr2=0x7fc6ac1fba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b0952700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6ac0b3370 0x7fc6ac1fba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b0952700 1 -- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6ac15bdd0 msgr2=0x7fc6ac1f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b0952700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6ac15bdd0 0x7fc6ac1f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b0952700 1 -- 172.21.15.89:0/3661937590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a4009b30 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b1153700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6ac0b3370 0x7fc6ac1fba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b0952700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac15c4b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c4e0 tx=0x7fc6a800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6a1ffb700 1 -- 172.21.15.89:0/3661937590 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a80052d0 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6a1ffb700 1 -- 172.21.15.89:0/3661937590 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a8005430 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac1fc380 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6a1ffb700 1 -- 172.21.15.89:0/3661937590 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a80038c0 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.723+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac15c9f0 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.724+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ac15dfe0 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.724+0000 7fc6a1ffb700 1 -- 172.21.15.89:0/3661937590 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc6a8003a20 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.725+0000 7fc6a1ffb700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc69006aff0 0x7fc69006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:14.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.725+0000 7fc6a1ffb700 1 -- 172.21.15.89:0/3661937590 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6a8088b50 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.725+0000 7fc6b1153700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc69006aff0 0x7fc69006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:14.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.725+0000 7fc6b1153700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc69006aff0 0x7fc69006d4b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc69c007a50 tx=0x7fc69c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:14.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.730+0000 7fc6a1ffb700 1 -- 172.21.15.89:0/3661937590 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6a8053af0 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.920+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6ac067f60 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.921+0000 7fc6a1ffb700 1 -- 172.21.15.89:0/3661937590 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16532 (secure 0 0 0) 0x7fc6a8017090 con 0x7fc6ac0d1c70 2023-11-24T06:21:14.924 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:14.924 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":64,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:58.434050+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:55.237439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:55.790741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:21:14.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc69006aff0 msgr2=0x7fc69006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc69006aff0 0x7fc69006d4b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc69c007a50 tx=0x7fc69c008040 comp rx=0 tx=0).stop 2023-11-24T06:21:14.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 msgr2=0x7fc6ac15c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:14.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac15c4b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c4e0 tx=0x7fc6a800c7f0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 shutdown_connections 2023-11-24T06:21:14.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc69006aff0 0x7fc69006d4b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6ac15bdd0 0x7fc6ac1f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6ac0b3370 0x7fc6ac1fba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 --2- 172.21.15.89:0/3661937590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6ac0d1c70 0x7fc6ac15c4b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:14.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 >> 172.21.15.89:0/3661937590 conn(0x7fc6ac0731a0 msgr2=0x7fc6ac0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:14.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 shutdown_connections 2023-11-24T06:21:14.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:14.923+0000 7fc6b33b7700 1 -- 172.21.15.89:0/3661937590 wait complete. 2023-11-24T06:21:14.938 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:20:55.237439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:20:55.790741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:21:14.939 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:21:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.058+0000 7ff1acadd700 1 Processor -- start 2023-11-24T06:21:15.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.058+0000 7ff1acadd700 1 -- start start 2023-11-24T06:21:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1acadd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff1a815e160 0x7ff1a80b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1acadd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff1a80746a0 0x7ff1a806d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1acadd700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a80b3c60 con 0x7ff1a80afc90 2023-11-24T06:21:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1acadd700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a80b4500 con 0x7ff1a815e160 2023-11-24T06:21:15.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1acadd700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a80b58c0 con 0x7ff1a80746a0 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1a659c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff1a815e160 0x7ff1a80b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1a659c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff1a815e160 0x7ff1a80b5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39336/0 (socket says 172.21.15.89:39336) 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1a659c700 1 -- 172.21.15.89:0/4087501677 learned_addr learned my addr 172.21.15.89:0/4087501677 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1a6d9d700 1 --2- 172.21.15.89:0/4087501677 >> v2:172.21.15.123:3300/0 conn(0x7ff1a80746a0 0x7ff1a806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1a659c700 1 -- 172.21.15.89:0/4087501677 >> v2:172.21.15.123:3300/0 conn(0x7ff1a80746a0 msgr2=0x7ff1a806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1a659c700 1 --2- 172.21.15.89:0/4087501677 >> v2:172.21.15.123:3300/0 conn(0x7ff1a80746a0 0x7ff1a806d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1a659c700 1 -- 172.21.15.89:0/4087501677 >> v1:172.21.15.123:6789/0 conn(0x7ff1a80afc90 legacy=0x7ff1a8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.059+0000 7ff1a659c700 1 -- 172.21.15.89:0/4087501677 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a80b0140 con 0x7ff1a815e160 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1a659c700 1 --2- 172.21.15.89:0/4087501677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a815e160 0x7ff1a80b5da0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff1a80d16e0 tx=0x7ff19c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e98b8bb5ed62dd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1a559a700 1 -- 172.21.15.89:0/4087501677 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c011830 con 0x7ff1a815e160 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1a559a700 1 -- 172.21.15.89:0/4087501677 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c011990 con 0x7ff1a815e160 2023-11-24T06:21:15.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1a559a700 1 -- 172.21.15.89:0/4087501677 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c011d20 con 0x7ff1a815e160 2023-11-24T06:21:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1acadd700 1 -- 172.21.15.89:0/4087501677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a815e160 msgr2=0x7ff1a80b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1acadd700 1 --2- 172.21.15.89:0/4087501677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a815e160 0x7ff1a80b5da0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff1a80d16e0 tx=0x7ff19c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1acadd700 1 -- 172.21.15.89:0/4087501677 shutdown_connections 2023-11-24T06:21:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1acadd700 1 --2- 172.21.15.89:0/4087501677 >> v2:172.21.15.123:3300/0 conn(0x7ff1a80746a0 0x7ff1a806d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1acadd700 1 --2- 172.21.15.89:0/4087501677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a815e160 0x7ff1a80b5da0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1acadd700 1 -- 172.21.15.89:0/4087501677 >> 172.21.15.89:0/4087501677 conn(0x7ff1a80d0bf0 msgr2=0x7ff1a80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1acadd700 1 -- 172.21.15.89:0/4087501677 shutdown_connections 2023-11-24T06:21:15.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.060+0000 7ff1acadd700 1 -- 172.21.15.89:0/4087501677 wait complete. 2023-11-24T06:21:15.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.062+0000 7ff1acadd700 1 Processor -- start 2023-11-24T06:21:15.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.062+0000 7ff1acadd700 1 -- start start 2023-11-24T06:21:15.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.062+0000 7ff1acadd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1a80746a0 0x7ff1a81f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1acadd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a80afc90 0x7ff1a80c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1acadd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1a815e160 0x7ff1a814a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1acadd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80c6110 con 0x7ff1a80afc90 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1acadd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a814cbc0 con 0x7ff1a815e160 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1acadd700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80c6320 con 0x7ff1a80746a0 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a80afc90 0x7ff1a80c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a80afc90 0x7ff1a80c5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39342/0 (socket says 172.21.15.89:39342) 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 -- 172.21.15.89:0/1143832445 learned_addr learned my addr 172.21.15.89:0/1143832445 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a659c700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1a80746a0 0x7ff1a81f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a6d9d700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1a815e160 0x7ff1a814a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 -- 172.21.15.89:0/1143832445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1a80746a0 msgr2=0x7ff1a81f7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1a80746a0 0x7ff1a81f7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 -- 172.21.15.89:0/1143832445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1a815e160 msgr2=0x7ff1a814a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1a815e160 0x7ff1a814a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 -- 172.21.15.89:0/1143832445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c009550 con 0x7ff1a80afc90 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1a5d9b700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a80afc90 0x7ff1a80c5bd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c4e0 tx=0x7ff19800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff196ffd700 1 -- 172.21.15.89:0/1143832445 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1980052d0 con 0x7ff1a80afc90 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff196ffd700 1 -- 172.21.15.89:0/1143832445 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff198005430 con 0x7ff1a80afc90 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff196ffd700 1 -- 172.21.15.89:0/1143832445 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1980038c0 con 0x7ff1a80afc90 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a80c6600 con 0x7ff1a80afc90 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.063+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a814a560 con 0x7ff1a80afc90 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.064+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a80c6b20 con 0x7ff1a80afc90 2023-11-24T06:21:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.065+0000 7ff196ffd700 1 -- 172.21.15.89:0/1143832445 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7ff198003a20 con 0x7ff1a80afc90 2023-11-24T06:21:15.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.065+0000 7ff196ffd700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff18806aff0 0x7ff18806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.065+0000 7ff196ffd700 1 -- 172.21.15.89:0/1143832445 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff198088a90 con 0x7ff1a80afc90 2023-11-24T06:21:15.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.065+0000 7ff1a659c700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff18806aff0 0x7ff18806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.066+0000 7ff1a659c700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff18806aff0 0x7ff18806d4b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff19c014040 tx=0x7ff19c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.070+0000 7ff196ffd700 1 -- 172.21.15.89:0/1143832445 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1a80c6b20 con 0x7ff1a80afc90 2023-11-24T06:21:15.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.261+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1a80c6b20 con 0x7ff1a80afc90 2023-11-24T06:21:15.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.261+0000 7ff196ffd700 1 -- 172.21.15.89:0/1143832445 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1a80c6b20 con 0x7ff1a80afc90 2023-11-24T06:21:15.264 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:21:15.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff18806aff0 msgr2=0x7ff18806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff18806aff0 0x7ff18806d4b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff19c014040 tx=0x7ff19c008030 comp rx=0 tx=0).stop 2023-11-24T06:21:15.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a80afc90 msgr2=0x7ff1a80c5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a80afc90 0x7ff1a80c5bd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c4e0 tx=0x7ff19800c7f0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 shutdown_connections 2023-11-24T06:21:15.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7ff18806aff0 0x7ff18806d4b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1a815e160 0x7ff1a814a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1a80746a0 0x7ff1a81f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 --2- 172.21.15.89:0/1143832445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a80afc90 0x7ff1a80c5bd0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 >> 172.21.15.89:0/1143832445 conn(0x7ff1a80d0bf0 msgr2=0x7ff1a815c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:15.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 shutdown_connections 2023-11-24T06:21:15.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.263+0000 7ff1acadd700 1 -- 172.21.15.89:0/1143832445 wait complete. 2023-11-24T06:21:15.278 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:21:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.398+0000 7f879a0cc700 1 Processor -- start 2023-11-24T06:21:15.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.398+0000 7f879a0cc700 1 -- start start 2023-11-24T06:21:15.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.399+0000 7f879a0cc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87940b5e40 0x7f879406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.399+0000 7f879a0cc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f879415f8f0 0x7f879415fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.399+0000 7f879a0cc700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879406b750 con 0x7f8794145720 2023-11-24T06:21:15.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.399+0000 7f879a0cc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8794160870 con 0x7f879415f8f0 2023-11-24T06:21:15.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.399+0000 7f879a0cc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8794160590 con 0x7f87940b5e40 2023-11-24T06:21:15.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.399+0000 7f8793fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f879415f8f0 0x7f879415fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.399+0000 7f87937fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8794145720 0x7f87940b5730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42852/0 (socket says 172.21.15.89:42852) 2023-11-24T06:21:15.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.399+0000 7f87937fe700 1 -- 172.21.15.89:0/1919699249 learned_addr learned my addr 172.21.15.89:0/1919699249 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:15.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f8792ffd700 1 --2- 172.21.15.89:0/1919699249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87940b5e40 0x7f879406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f8793fff700 1 -- 172.21.15.89:0/1919699249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87940b5e40 msgr2=0x7f879406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:15.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f8793fff700 1 --2- 172.21.15.89:0/1919699249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87940b5e40 0x7f879406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f8793fff700 1 -- 172.21.15.89:0/1919699249 >> v1:172.21.15.89:6789/0 conn(0x7f8794145720 legacy=0x7f87940b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f87927fc700 1 -- 172.21.15.89:0/1919699249 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286426351 0 0) 0x7f879406b750 con 0x7f8794145720 2023-11-24T06:21:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f8793fff700 1 -- 172.21.15.89:0/1919699249 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879406b100 con 0x7f879415f8f0 2023-11-24T06:21:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f8793fff700 1 --2- 172.21.15.89:0/1919699249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879415f8f0 0x7f879415fd60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f878c009d90 tx=0x7f878c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da74bac9e156ccf6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f87927fc700 1 -- 172.21.15.89:0/1919699249 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c00e070 con 0x7f879415f8f0 2023-11-24T06:21:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f87927fc700 1 -- 172.21.15.89:0/1919699249 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c00bbd0 con 0x7f879415f8f0 2023-11-24T06:21:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f87927fc700 1 -- 172.21.15.89:0/1919699249 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c012430 con 0x7f879415f8f0 2023-11-24T06:21:15.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f879a0cc700 1 -- 172.21.15.89:0/1919699249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879415f8f0 msgr2=0x7f879415fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.400+0000 7f879a0cc700 1 --2- 172.21.15.89:0/1919699249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879415f8f0 0x7f879415fd60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f878c009d90 tx=0x7f878c00b330 comp rx=0 tx=0).stop 2023-11-24T06:21:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.401+0000 7f879a0cc700 1 -- 172.21.15.89:0/1919699249 shutdown_connections 2023-11-24T06:21:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.401+0000 7f879a0cc700 1 --2- 172.21.15.89:0/1919699249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f879415f8f0 0x7f879415fd60 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.401+0000 7f879a0cc700 1 --2- 172.21.15.89:0/1919699249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87940b5e40 0x7f879406bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.401+0000 7f879a0cc700 1 -- 172.21.15.89:0/1919699249 >> 172.21.15.89:0/1919699249 conn(0x7f87940d0be0 msgr2=0x7f87940d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:15.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.401+0000 7f879a0cc700 1 -- 172.21.15.89:0/1919699249 shutdown_connections 2023-11-24T06:21:15.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.401+0000 7f879a0cc700 1 -- 172.21.15.89:0/1919699249 wait complete. 2023-11-24T06:21:15.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.402+0000 7f879a0cc700 1 Processor -- start 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.402+0000 7f879a0cc700 1 -- start start 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f879a0cc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87940b5e40 0x7f87941f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f879a0cc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8794145720 0x7f87941433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f879a0cc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879415f8f0 0x7f879406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f879a0cc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8794143ea0 con 0x7f87940b5e40 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f879a0cc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87941f7d30 con 0x7f8794145720 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f879a0cc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879413f5f0 con 0x7f879415f8f0 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87940b5e40 0x7f87941f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87940b5e40 0x7f87941f3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39354/0 (socket says 172.21.15.89:39354) 2023-11-24T06:21:15.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 -- 172.21.15.89:0/100926745 learned_addr learned my addr 172.21.15.89:0/100926745 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 -- 172.21.15.89:0/100926745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879415f8f0 msgr2=0x7f879406e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879415f8f0 0x7f879406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 -- 172.21.15.89:0/100926745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8794145720 msgr2=0x7f87941433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8794145720 0x7f87941433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 -- 172.21.15.89:0/100926745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c009a70 con 0x7f87940b5e40 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f87937fe700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87940b5e40 0x7f87941f3010 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f878800c750 tx=0x7f878800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f877bfff700 1 -- 172.21.15.89:0/100926745 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788004c80 con 0x7f87940b5e40 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f877bfff700 1 -- 172.21.15.89:0/100926745 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8788004e00 con 0x7f87940b5e40 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87941f6f70 con 0x7f87940b5e40 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f877bfff700 1 -- 172.21.15.89:0/100926745 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788007310 con 0x7f87940b5e40 2023-11-24T06:21:15.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.403+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87941f7530 con 0x7f87940b5e40 2023-11-24T06:21:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.404+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8794068fb0 con 0x7f87940b5e40 2023-11-24T06:21:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.405+0000 7f877bfff700 1 -- 172.21.15.89:0/100926745 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8788007470 con 0x7f87940b5e40 2023-11-24T06:21:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.405+0000 7f877bfff700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f878006aff0 0x7f878006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.405+0000 7f877bfff700 1 -- 172.21.15.89:0/100926745 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87880894b0 con 0x7f87940b5e40 2023-11-24T06:21:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.405+0000 7f8792ffd700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f878006aff0 0x7f878006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.405+0000 7f8792ffd700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f878006aff0 0x7f878006d4b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8794144650 tx=0x7f8784008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.409+0000 7f877bfff700 1 -- 172.21.15.89:0/100926745 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8788054450 con 0x7f87940b5e40 2023-11-24T06:21:15.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.598+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f879406c140 con 0x7f87940b5e40 2023-11-24T06:21:15.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.598+0000 7f877bfff700 1 -- 172.21.15.89:0/100926745 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v64) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8788057a70 con 0x7f87940b5e40 2023-11-24T06:21:15.602 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f878006aff0 msgr2=0x7f878006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f878006aff0 0x7f878006d4b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8794144650 tx=0x7f8784008040 comp rx=0 tx=0).stop 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87940b5e40 msgr2=0x7f87941f3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87940b5e40 0x7f87941f3010 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f878800c750 tx=0x7f878800cb10 comp rx=0 tx=0).stop 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 shutdown_connections 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f879415f8f0 0x7f879406e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87940b5e40 0x7f87941f3010 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f878006aff0 0x7f878006d4b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 --2- 172.21.15.89:0/100926745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8794145720 0x7f87941433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.600+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 >> 172.21.15.89:0/100926745 conn(0x7f87940d0be0 msgr2=0x7f87940b01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:15.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.601+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 shutdown_connections 2023-11-24T06:21:15.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.601+0000 7f879a0cc700 1 -- 172.21.15.89:0/100926745 wait complete. 2023-11-24T06:21:15.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:21:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.737+0000 7f9ce7fcb700 1 Processor -- start 2023-11-24T06:21:15.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.737+0000 7f9ce7fcb700 1 -- start start 2023-11-24T06:21:15.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce7fcb700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9ce00cf610 0x7f9ce00add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce7fcb700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ce0073f90 0x7f9ce0074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce7fcb700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce006b080 con 0x7f9ce015f9e0 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce7fcb700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce0074960 con 0x7f9ce0073f90 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce7fcb700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce015c290 con 0x7f9ce00cf610 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce6568700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ce0073f90 0x7f9ce0074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce6568700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ce0073f90 0x7f9ce0074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39366/0 (socket says 172.21.15.89:39366) 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce6568700 1 -- 172.21.15.89:0/2912508417 learned_addr learned my addr 172.21.15.89:0/2912508417 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce5d67700 1 --2- 172.21.15.89:0/2912508417 >> v2:172.21.15.123:3300/0 conn(0x7f9ce00cf610 0x7f9ce00add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce6568700 1 -- 172.21.15.89:0/2912508417 >> v2:172.21.15.123:3300/0 conn(0x7f9ce00cf610 msgr2=0x7f9ce00add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:15.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce6568700 1 --2- 172.21.15.89:0/2912508417 >> v2:172.21.15.123:3300/0 conn(0x7f9ce00cf610 0x7f9ce00add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce6568700 1 -- 172.21.15.89:0/2912508417 >> v1:172.21.15.123:6789/0 conn(0x7f9ce015f9e0 legacy=0x7f9ce00b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.738+0000 7f9ce6568700 1 -- 172.21.15.89:0/2912508417 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce00cf9f0 con 0x7f9ce0073f90 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce6568700 1 --2- 172.21.15.89:0/2912508417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0074420 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009e50 tx=0x7f9cd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d20991e99c88616 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce4d65700 1 -- 172.21.15.89:0/2912508417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd800e070 con 0x7f9ce0073f90 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce4d65700 1 -- 172.21.15.89:0/2912508417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd8003a20 con 0x7f9ce0073f90 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce4d65700 1 -- 172.21.15.89:0/2912508417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8003dd0 con 0x7f9ce0073f90 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2912508417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 msgr2=0x7f9ce0074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2912508417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0074420 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009e50 tx=0x7f9cd800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2912508417 shutdown_connections 2023-11-24T06:21:15.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2912508417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0074420 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2912508417 >> v2:172.21.15.123:3300/0 conn(0x7f9ce00cf610 0x7f9ce00add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2912508417 >> 172.21.15.89:0/2912508417 conn(0x7f9ce00d0bf0 msgr2=0x7f9ce00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:15.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2912508417 shutdown_connections 2023-11-24T06:21:15.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.739+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2912508417 wait complete. 2023-11-24T06:21:15.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.741+0000 7f9ce7fcb700 1 Processor -- start 2023-11-24T06:21:15.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.741+0000 7f9ce7fcb700 1 -- start start 2023-11-24T06:21:15.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce7fcb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0140210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce7fcb700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ce00cf610 0x7f9ce013fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce7fcb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce015f9e0 0x7f9ce0142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce7fcb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce0156170 con 0x7f9ce0073f90 2023-11-24T06:21:15.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce7fcb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce013ff50 con 0x7f9ce015f9e0 2023-11-24T06:21:15.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce7fcb700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce0154690 con 0x7f9ce00cf610 2023-11-24T06:21:15.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0140210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0140210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39368/0 (socket says 172.21.15.89:39368) 2023-11-24T06:21:15.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 -- 172.21.15.89:0/2672665124 learned_addr learned my addr 172.21.15.89:0/2672665124 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:15.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 -- 172.21.15.89:0/2672665124 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ce00cf610 msgr2=0x7f9ce013fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ce00cf610 0x7f9ce013fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 -- 172.21.15.89:0/2672665124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce015f9e0 msgr2=0x7f9ce0142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce015f9e0 0x7f9ce0142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 -- 172.21.15.89:0/2672665124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd8009b30 con 0x7f9ce0073f90 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.742+0000 7f9ce5d67700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0140210 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000e810 tx=0x7f9cd000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.743+0000 7f9cd67fc700 1 -- 172.21.15.89:0/2672665124 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0013610 con 0x7f9ce0073f90 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.743+0000 7f9cd67fc700 1 -- 172.21.15.89:0/2672665124 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd0013770 con 0x7f9ce0073f90 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.743+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce01431b0 con 0x7f9ce0073f90 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.743+0000 7f9cd67fc700 1 -- 172.21.15.89:0/2672665124 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0013b20 con 0x7f9ce0073f90 2023-11-24T06:21:15.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.743+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce0156380 con 0x7f9ce0073f90 2023-11-24T06:21:15.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.744+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce0068fb0 con 0x7f9ce0073f90 2023-11-24T06:21:15.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.744+0000 7f9cd67fc700 1 -- 172.21.15.89:0/2672665124 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f9cd0013c80 con 0x7f9ce0073f90 2023-11-24T06:21:15.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.744+0000 7f9cd67fc700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cc406aff0 0x7f9cc406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:15.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.744+0000 7f9ce5566700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cc406aff0 0x7f9cc406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:15.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.744+0000 7f9cd67fc700 1 -- 172.21.15.89:0/2672665124 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cd0089530 con 0x7f9ce0073f90 2023-11-24T06:21:15.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.745+0000 7f9ce5566700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cc406aff0 0x7f9cc406d4b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00b2440 tx=0x7f9cdc007a40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:15.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.749+0000 7f9cd67fc700 1 -- 172.21.15.89:0/2672665124 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cd00543d0 con 0x7f9ce0073f90 2023-11-24T06:21:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.940+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9ce015bb20 con 0x7f9ce0073f90 2023-11-24T06:21:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.940+0000 7f9cd67fc700 1 -- 172.21.15.89:0/2672665124 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v64) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9cd00579f0 con 0x7f9ce0073f90 2023-11-24T06:21:15.943 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:21:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cc406aff0 msgr2=0x7f9cc406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cc406aff0 0x7f9cc406d4b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00b2440 tx=0x7f9cdc007a40 comp rx=0 tx=0).stop 2023-11-24T06:21:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 msgr2=0x7f9ce0140210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0140210 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000e810 tx=0x7f9cd000ebd0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 shutdown_connections 2023-11-24T06:21:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f9cc406aff0 0x7f9cc406d4b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce015f9e0 0x7f9ce0142c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ce0073f90 0x7f9ce0140210 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 --2- 172.21.15.89:0/2672665124 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ce00cf610 0x7f9ce013fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 >> 172.21.15.89:0/2672665124 conn(0x7f9ce00d0bf0 msgr2=0x7f9ce00b0930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 shutdown_connections 2023-11-24T06:21:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:15.942+0000 7f9ce7fcb700 1 -- 172.21.15.89:0/2672665124 wait complete. 2023-11-24T06:21:15.957 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:21:16.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.077+0000 7f8eceb8c700 1 Processor -- start 2023-11-24T06:21:16.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.077+0000 7f8eceb8c700 1 -- start start 2023-11-24T06:21:16.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8eceb8c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8ec815e080 0x7f8ec80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8eceb8c700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec80ae430 con 0x7f8ec806c710 2023-11-24T06:21:16.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8eceb8c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec80ae710 con 0x7f8ec80af4d0 2023-11-24T06:21:16.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8eceb8c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec80af0b0 con 0x7f8ec815e080 2023-11-24T06:21:16.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ecd129700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8ec806c710 0x7f8ec806e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42884/0 (socket says 172.21.15.89:42884) 2023-11-24T06:21:16.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ecd129700 1 -- 172.21.15.89:0/1128355426 learned_addr learned my addr 172.21.15.89:0/1128355426 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:16.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ecc928700 1 --2- 172.21.15.89:0/1128355426 >> v2:172.21.15.123:3300/0 conn(0x7f8ec815e080 0x7f8ec80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581161410 0 0) 0x7f8ec80ae430 con 0x7f8ec806c710 2023-11-24T06:21:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb80035b0 con 0x7f8ec806c710 2023-11-24T06:21:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3894668813 0 0) 0x7f8eb80035b0 con 0x7f8ec806c710 2023-11-24T06:21:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 >> v2:172.21.15.123:3300/0 conn(0x7f8ec815e080 msgr2=0x7f8ec80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 --2- 172.21.15.89:0/1128355426 >> v2:172.21.15.123:3300/0 conn(0x7f8ec815e080 0x7f8ec80b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 >> v1:172.21.15.123:6789/0 conn(0x7f8ec80af4d0 legacy=0x7f8ec806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec80ae9f0 con 0x7f8ec806c710 2023-11-24T06:21:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f8ec00019f0 con 0x7f8ec806c710 2023-11-24T06:21:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.078+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 >> v1:172.21.15.89:6789/0 conn(0x7f8ec806c710 legacy=0x7f8ec806e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb8003a00 0x7f8eb8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8eb8004850 0x7f8eb8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8005210 0x7f8eb800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80af0b0 con 0x7f8eb8003a00 2023-11-24T06:21:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb800c0b0 con 0x7f8eb8004850 2023-11-24T06:21:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb800c270 con 0x7f8eb8005210 2023-11-24T06:21:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecc928700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb8003a00 0x7f8eb8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec7fff700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8eb8004850 0x7f8eb8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecd129700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8005210 0x7f8eb800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecc928700 1 -- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8005210 msgr2=0x7f8eb800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecc928700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8005210 0x7f8eb800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecc928700 1 -- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8eb8004850 msgr2=0x7f8eb8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecc928700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8eb8004850 0x7f8eb8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecc928700 1 -- 172.21.15.89:0/1128355426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb80035b0 con 0x7f8eb8003a00 2023-11-24T06:21:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecd129700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8005210 0x7f8eb800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecd129700 1 -- 172.21.15.89:0/1128355426 reap_dead start 2023-11-24T06:21:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ecc928700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb8003a00 0x7f8eb8003e20 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009b60 tx=0x7f8eb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740ccad0977d110d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc00e6c0 con 0x7f8eb8003a00 2023-11-24T06:21:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc00e820 con 0x7f8eb8003a00 2023-11-24T06:21:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.079+0000 7f8ec77fe700 1 -- 172.21.15.89:0/1128355426 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc00eb90 con 0x7f8eb8003a00 2023-11-24T06:21:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.080+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1128355426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb8003a00 msgr2=0x7f8eb8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.080+0000 7f8eceb8c700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb8003a00 0x7f8eb8003e20 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009b60 tx=0x7f8eb8005880 comp rx=0 tx=0).stop 2023-11-24T06:21:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.080+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1128355426 shutdown_connections 2023-11-24T06:21:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.080+0000 7f8eceb8c700 1 --2- 172.21.15.89:0/1128355426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb8003a00 0x7f8eb8003e20 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.080+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1128355426 >> 172.21.15.89:0/1128355426 conn(0x7f8ec8071b90 msgr2=0x7f8ec8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.080+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1128355426 shutdown_connections 2023-11-24T06:21:16.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.080+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1128355426 wait complete. 2023-11-24T06:21:16.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.081+0000 7f8eceb8c700 1 Processor -- start 2023-11-24T06:21:16.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.081+0000 7f8eceb8c700 1 -- start start 2023-11-24T06:21:16.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8eceb8c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8003a00 0x7f8ec815c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8eceb8c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ec8151850 0x7f8ec8158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8eceb8c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8150b40 0x7f8ec81595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8eceb8c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec8157250 con 0x7f8ec8151850 2023-11-24T06:21:16.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8eceb8c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec8158a40 con 0x7f8ec8150b40 2023-11-24T06:21:16.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8eceb8c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec8157460 con 0x7f8eb8003a00 2023-11-24T06:21:16.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ec8151850 0x7f8ec8158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ec8151850 0x7f8ec8158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39390/0 (socket says 172.21.15.89:39390) 2023-11-24T06:21:16.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 -- 172.21.15.89:0/1720948236 learned_addr learned my addr 172.21.15.89:0/1720948236 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:16.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ecc928700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8003a00 0x7f8ec815c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ecd129700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8150b40 0x7f8ec81595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 -- 172.21.15.89:0/1720948236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8003a00 msgr2=0x7f8ec815c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8003a00 0x7f8ec815c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 -- 172.21.15.89:0/1720948236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8150b40 msgr2=0x7f8ec81595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8150b40 0x7f8ec81595d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 -- 172.21.15.89:0/1720948236 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec8157740 con 0x7f8ec8151850 2023-11-24T06:21:16.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.082+0000 7f8ec7fff700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ec8151850 0x7f8ec8158500 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400f820 tx=0x7f8eb400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.083+0000 7f8ec57fa700 1 -- 172.21.15.89:0/1720948236 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40052e0 con 0x7f8ec8151850 2023-11-24T06:21:16.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.083+0000 7f8ec57fa700 1 -- 172.21.15.89:0/1720948236 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb4005440 con 0x7f8ec8151850 2023-11-24T06:21:16.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.083+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec8157a20 con 0x7f8ec8151850 2023-11-24T06:21:16.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.083+0000 7f8ec57fa700 1 -- 172.21.15.89:0/1720948236 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40038a0 con 0x7f8ec8151850 2023-11-24T06:21:16.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.083+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec8157fe0 con 0x7f8ec8151850 2023-11-24T06:21:16.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.084+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec81581a0 con 0x7f8ec8151850 2023-11-24T06:21:16.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.084+0000 7f8ec57fa700 1 -- 172.21.15.89:0/1720948236 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8eb4003a00 con 0x7f8ec8151850 2023-11-24T06:21:16.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.085+0000 7f8ec57fa700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eac06aff0 0x7f8eac06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.085+0000 7f8ecc928700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eac06aff0 0x7f8eac06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.085+0000 7f8ec57fa700 1 -- 172.21.15.89:0/1720948236 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8eb4089540 con 0x7f8ec8151850 2023-11-24T06:21:16.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.085+0000 7f8ecc928700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eac06aff0 0x7f8eac06d4b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0090b0 tx=0x7f8ebc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.089+0000 7f8ec57fa700 1 -- 172.21.15.89:0/1720948236 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8eb40544e0 con 0x7f8ec8151850 2023-11-24T06:21:16.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.297+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8ec80753f0 con 0x7f8ec8151850 2023-11-24T06:21:16.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.297+0000 7f8ec57fa700 1 -- 172.21.15.89:0/1720948236 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v64) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8eb4003cb0 con 0x7f8ec8151850 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eac06aff0 msgr2=0x7f8eac06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eac06aff0 0x7f8eac06d4b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0090b0 tx=0x7f8ebc008030 comp rx=0 tx=0).stop 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ec8151850 msgr2=0x7f8ec8158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ec8151850 0x7f8ec8158500 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400f820 tx=0x7f8eb400fbe0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 shutdown_connections 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f8eac06aff0 0x7f8eac06d4b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8150b40 0x7f8ec81595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8eb8003a00 0x7f8ec815c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 --2- 172.21.15.89:0/1720948236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ec8151850 0x7f8ec8158500 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 >> 172.21.15.89:0/1720948236 conn(0x7f8ec8071b90 msgr2=0x7f8ec80b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.299+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 shutdown_connections 2023-11-24T06:21:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.300+0000 7f8eceb8c700 1 -- 172.21.15.89:0/1720948236 wait complete. 2023-11-24T06:21:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2023-11-24T06:21:16.315 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T06:21:16.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:16.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T06:21:16.366 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:21:16.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:16.366 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T06:21:16.387 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:21:16.387 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:16.405 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:21:16.406 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:21:16.406 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:16.447 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:21:16.448 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:21:16.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.565+0000 7fc8c97d1700 1 Processor -- start 2023-11-24T06:21:16.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.565+0000 7fc8c97d1700 1 -- start start 2023-11-24T06:21:16.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c97d1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c415de60 0x7fc8c40ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c97d1700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c40b99e0 con 0x7fc8c40d4410 2023-11-24T06:21:16.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c97d1700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c40d1250 con 0x7fc8c40ced60 2023-11-24T06:21:16.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c97d1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40d1530 con 0x7fc8c415de60 2023-11-24T06:21:16.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c37fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc8c40d4410 0x7fc8c40d1940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42902/0 (socket says 172.21.15.89:42902) 2023-11-24T06:21:16.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c37fe700 1 -- 172.21.15.89:0/3313726780 learned_addr learned my addr 172.21.15.89:0/3313726780 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:16.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230164397 0 0) 0x7fc8c40b99e0 con 0x7fc8c40d4410 2023-11-24T06:21:16.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c2ffd700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c415de60 0x7fc8c40ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.566+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ac0035b0 con 0x7fc8c40d4410 2023-11-24T06:21:16.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147349206 0 0) 0x7fc8ac0035b0 con 0x7fc8c40d4410 2023-11-24T06:21:16.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c415de60 msgr2=0x7fc8c40ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:16.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c415de60 0x7fc8c40ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 >> v1:172.21.15.123:6789/0 conn(0x7fc8c40ced60 legacy=0x7fc8c40d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40d33f0 con 0x7fc8c40d4410 2023-11-24T06:21:16.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fc8bc0019f0 con 0x7fc8c40d4410 2023-11-24T06:21:16.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 >> v1:172.21.15.89:6789/0 conn(0x7fc8c40d4410 legacy=0x7fc8c40d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac003a30 0x7fc8ac003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ac004850 0x7fc8ac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c415de60 0x7fc8ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40d1530 con 0x7fc8ac004850 2023-11-24T06:21:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac00bac0 con 0x7fc8ac003a30 2023-11-24T06:21:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac00bc80 con 0x7fc8c415de60 2023-11-24T06:21:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c27fc700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ac004850 0x7fc8ac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c2ffd700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac003a30 0x7fc8ac003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c37fe700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c415de60 0x7fc8ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c27fc700 1 -- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c415de60 msgr2=0x7fc8ac005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c27fc700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c415de60 0x7fc8ac005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c27fc700 1 -- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac003a30 msgr2=0x7fc8ac003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c27fc700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac003a30 0x7fc8ac003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c27fc700 1 -- 172.21.15.89:0/3313726780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac0035b0 con 0x7fc8ac004850 2023-11-24T06:21:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c27fc700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ac004850 0x7fc8ac004cd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8007fd0 tx=0x7fc8ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf73766c6cfc926f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80052d0 con 0x7fc8ac004850 2023-11-24T06:21:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b8005470 con 0x7fc8ac004850 2023-11-24T06:21:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.567+0000 7fc8c1ffb700 1 -- 172.21.15.89:0/3313726780 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80038c0 con 0x7fc8ac004850 2023-11-24T06:21:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.568+0000 7fc8c97d1700 1 -- 172.21.15.89:0/3313726780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ac004850 msgr2=0x7fc8ac004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.568+0000 7fc8c97d1700 1 --2- 172.21.15.89:0/3313726780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8ac004850 0x7fc8ac004cd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8007fd0 tx=0x7fc8ac005880 comp rx=0 tx=0).stop 2023-11-24T06:21:16.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.568+0000 7fc8c37fe700 1 -- 172.21.15.89:0/3313726780 reap_dead start 2023-11-24T06:21:16.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.568+0000 7fc8c97d1700 1 -- 172.21.15.89:0/3313726780 shutdown_connections 2023-11-24T06:21:16.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.568+0000 7fc8c97d1700 1 -- 172.21.15.89:0/3313726780 >> 172.21.15.89:0/3313726780 conn(0x7fc8c406b410 msgr2=0x7fc8c406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:16.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.568+0000 7fc8c97d1700 1 -- 172.21.15.89:0/3313726780 shutdown_connections 2023-11-24T06:21:16.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.568+0000 7fc8c97d1700 1 -- 172.21.15.89:0/3313726780 wait complete. 2023-11-24T06:21:16.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.569+0000 7fc8c97d1700 1 Processor -- start 2023-11-24T06:21:16.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.569+0000 7fc8c97d1700 1 -- start start 2023-11-24T06:21:16.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c97d1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac004850 0x7fc8c40cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c97d1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c40cc240 0x7fc8c4158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c97d1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c415eaa0 0x7fc8c40b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c97d1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4158bb0 con 0x7fc8c415eaa0 2023-11-24T06:21:16.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c97d1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40cc800 con 0x7fc8ac004850 2023-11-24T06:21:16.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c97d1700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40ccae0 con 0x7fc8c40cc240 2023-11-24T06:21:16.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c415eaa0 0x7fc8c40b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c415eaa0 0x7fc8c40b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39410/0 (socket says 172.21.15.89:39410) 2023-11-24T06:21:16.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c37fe700 1 -- 172.21.15.89:0/2054178597 learned_addr learned my addr 172.21.15.89:0/2054178597 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:16.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c2ffd700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac004850 0x7fc8c40cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.570+0000 7fc8c27fc700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c40cc240 0x7fc8c4158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8c37fe700 1 -- 172.21.15.89:0/2054178597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c40cc240 msgr2=0x7fc8c4158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8c37fe700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c40cc240 0x7fc8c4158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8c37fe700 1 -- 172.21.15.89:0/2054178597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac004850 msgr2=0x7fc8c40cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8c37fe700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac004850 0x7fc8c40cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8c37fe700 1 -- 172.21.15.89:0/2054178597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40ccdc0 con 0x7fc8c415eaa0 2023-11-24T06:21:16.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8c37fe700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c415eaa0 0x7fc8c40b4530 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40d01c0 tx=0x7fc8bc00a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8ab7fe700 1 -- 172.21.15.89:0/2054178597 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc00acb0 con 0x7fc8c415eaa0 2023-11-24T06:21:16.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8ab7fe700 1 -- 172.21.15.89:0/2054178597 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc00ae10 con 0x7fc8c415eaa0 2023-11-24T06:21:16.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c40cd0a0 con 0x7fc8c415eaa0 2023-11-24T06:21:16.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8ab7fe700 1 -- 172.21.15.89:0/2054178597 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc013630 con 0x7fc8c415eaa0 2023-11-24T06:21:16.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.571+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c415ef20 con 0x7fc8c415eaa0 2023-11-24T06:21:16.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.572+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c4068f10 con 0x7fc8c415eaa0 2023-11-24T06:21:16.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.572+0000 7fc8ab7fe700 1 -- 172.21.15.89:0/2054178597 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc8bc013790 con 0x7fc8c415eaa0 2023-11-24T06:21:16.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.573+0000 7fc8ab7fe700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8b006aff0 0x7fc8b006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.573+0000 7fc8ab7fe700 1 -- 172.21.15.89:0/2054178597 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8bc08a230 con 0x7fc8c415eaa0 2023-11-24T06:21:16.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.573+0000 7fc8c2ffd700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8b006aff0 0x7fc8b006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.573+0000 7fc8c2ffd700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8b006aff0 0x7fc8b006d4b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009be0 tx=0x7fc8b4009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.577+0000 7fc8ab7fe700 1 -- 172.21.15.89:0/2054178597 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8bc055320 con 0x7fc8c415eaa0 2023-11-24T06:21:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.809+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8c406bc10 con 0x7fc8c415eaa0 2023-11-24T06:21:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.810+0000 7fc8ab7fe700 1 -- 172.21.15.89:0/2054178597 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fc8bc058940 con 0x7fc8c415eaa0 2023-11-24T06:21:16.813 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:16.813 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/351443905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":351443905},{"type":"v1","addr":"172.21.15.123:6834","nonce":351443905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6021,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2890850053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2890850053},{"type":"v1","addr":"172.21.15.89:6836","nonce":2890850053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3912248946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3912248946},{"type":"v1","addr":"172.21.15.89:6837","nonce":3912248946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:56.834302+0000","modified":"2023-11-24T06:20:59.846940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6039":{"gid":6039,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2963752713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2963752713},{"type":"v1","addr":"172.21.15.123:6835","nonce":2963752713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T06:21:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8b006aff0 msgr2=0x7fc8b006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8b006aff0 0x7fc8b006d4b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009be0 tx=0x7fc8b4009210 comp rx=0 tx=0).stop 2023-11-24T06:21:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c415eaa0 msgr2=0x7fc8c40b4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c415eaa0 0x7fc8c40b4530 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40d01c0 tx=0x7fc8bc00a5f0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 shutdown_connections 2023-11-24T06:21:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8c415eaa0 0x7fc8c40b4530 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8b006aff0 0x7fc8b006d4b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8ac004850 0x7fc8c40cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 --2- 172.21.15.89:0/2054178597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8c40cc240 0x7fc8c4158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 >> 172.21.15.89:0/2054178597 conn(0x7fc8c406b410 msgr2=0x7fc8c40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 shutdown_connections 2023-11-24T06:21:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.812+0000 7fc8c97d1700 1 -- 172.21.15.89:0/2054178597 wait complete. 2023-11-24T06:21:16.818 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 42 2023-11-24T06:21:16.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:21:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.944+0000 7f025da38700 1 Processor -- start 2023-11-24T06:21:16.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.944+0000 7f025da38700 1 -- start start 2023-11-24T06:21:16.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.945+0000 7f025da38700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f025815e080 0x7f02580b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.945+0000 7f025da38700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f025806c710 0x7f02580afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.945+0000 7f025da38700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580ae430 con 0x7f02580af4d0 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.945+0000 7f025da38700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02580af0b0 con 0x7f025815e080 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.945+0000 7f025da38700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02580b05a0 con 0x7f025806c710 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.945+0000 7f0256ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f025815e080 0x7f02580b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.945+0000 7f0256ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f025815e080 0x7f02580b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39416/0 (socket says 172.21.15.89:39416) 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.945+0000 7f0256ffd700 1 -- 172.21.15.89:0/2845244307 learned_addr learned my addr 172.21.15.89:0/2845244307 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f02577fe700 1 --2- 172.21.15.89:0/2845244307 >> v2:172.21.15.123:3300/0 conn(0x7f025806c710 0x7f02580afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f0256ffd700 1 -- 172.21.15.89:0/2845244307 >> v2:172.21.15.123:3300/0 conn(0x7f025806c710 msgr2=0x7f02580afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f0256ffd700 1 --2- 172.21.15.89:0/2845244307 >> v2:172.21.15.123:3300/0 conn(0x7f025806c710 0x7f02580afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f0256ffd700 1 -- 172.21.15.89:0/2845244307 >> v1:172.21.15.123:6789/0 conn(0x7f02580af4d0 legacy=0x7f025806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f0256ffd700 1 -- 172.21.15.89:0/2845244307 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580ae710 con 0x7f025815e080 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f0256ffd700 1 --2- 172.21.15.89:0/2845244307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f02580b5050 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f025806e060 tx=0x7f024c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a4d069a04c93ba server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f0255ffb700 1 -- 172.21.15.89:0/2845244307 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c004230 con 0x7f025815e080 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f0255ffb700 1 -- 172.21.15.89:0/2845244307 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024c004390 con 0x7f025815e080 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f0255ffb700 1 -- 172.21.15.89:0/2845244307 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c0036a0 con 0x7f025815e080 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f025da38700 1 -- 172.21.15.89:0/2845244307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 msgr2=0x7f02580b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f025da38700 1 --2- 172.21.15.89:0/2845244307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f02580b5050 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f025806e060 tx=0x7f024c009c00 comp rx=0 tx=0).stop 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f025da38700 1 -- 172.21.15.89:0/2845244307 shutdown_connections 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f025da38700 1 --2- 172.21.15.89:0/2845244307 >> v2:172.21.15.123:3300/0 conn(0x7f025806c710 0x7f02580afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f025da38700 1 --2- 172.21.15.89:0/2845244307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f02580b5050 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.946+0000 7f025da38700 1 -- 172.21.15.89:0/2845244307 >> 172.21.15.89:0/2845244307 conn(0x7f0258071b90 msgr2=0x7f0258071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:16.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.947+0000 7f025da38700 1 -- 172.21.15.89:0/2845244307 shutdown_connections 2023-11-24T06:21:16.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.947+0000 7f025da38700 1 -- 172.21.15.89:0/2845244307 wait complete. 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.948+0000 7f025da38700 1 Processor -- start 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.948+0000 7f025da38700 1 -- start start 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f025da38700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f025806c710 0x7f025813eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f025da38700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02580af4d0 0x7f025813f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f025da38700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f0258155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f025da38700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025813f0c0 con 0x7f025815e080 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f025da38700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025814e680 con 0x7f025806c710 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f025da38700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025815a210 con 0x7f02580af4d0 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f0258155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f0258155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39424/0 (socket says 172.21.15.89:39424) 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 -- 172.21.15.89:0/4252531303 learned_addr learned my addr 172.21.15.89:0/4252531303 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:16.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f0256ffd700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f025806c710 0x7f025813eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02567fc700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02580af4d0 0x7f025813f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 -- 172.21.15.89:0/4252531303 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02580af4d0 msgr2=0x7f025813f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02580af4d0 0x7f025813f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 -- 172.21.15.89:0/4252531303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f025806c710 msgr2=0x7f025813eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f025806c710 0x7f025813eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 -- 172.21.15.89:0/4252531303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c009550 con 0x7f025815e080 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f02577fe700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f0258155fa0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f025000cd40 tx=0x7f025000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f023ffff700 1 -- 172.21.15.89:0/4252531303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250003a20 con 0x7f025815e080 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f023ffff700 1 -- 172.21.15.89:0/4252531303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0250003b80 con 0x7f025815e080 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025815a4f0 con 0x7f025815e080 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f023ffff700 1 -- 172.21.15.89:0/4252531303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250004020 con 0x7f025815e080 2023-11-24T06:21:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.949+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0258155a80 con 0x7f025815e080 2023-11-24T06:21:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.950+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02580753f0 con 0x7f025815e080 2023-11-24T06:21:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.951+0000 7f023ffff700 1 -- 172.21.15.89:0/4252531303 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0250004180 con 0x7f025815e080 2023-11-24T06:21:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.951+0000 7f023ffff700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f024406aff0 0x7f024406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.951+0000 7f023ffff700 1 -- 172.21.15.89:0/4252531303 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0250012070 con 0x7f025815e080 2023-11-24T06:21:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.951+0000 7f0256ffd700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f024406aff0 0x7f024406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.951+0000 7f0256ffd700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f024406aff0 0x7f024406d4b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f024c009c40 tx=0x7f024c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:16.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:16.955+0000 7f023ffff700 1 -- 172.21.15.89:0/4252531303 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0250054f70 con 0x7f025815e080 2023-11-24T06:21:17.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.148+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0258069050 con 0x7f025815e080 2023-11-24T06:21:17.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.149+0000 7f023ffff700 1 -- 172.21.15.89:0/4252531303 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f0250058590 con 0x7f025815e080 2023-11-24T06:21:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:17.152 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":64,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:58.434050+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:55.237439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:55.790741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:21:17.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f024406aff0 msgr2=0x7f024406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f024406aff0 0x7f024406d4b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f024c009c40 tx=0x7f024c008030 comp rx=0 tx=0).stop 2023-11-24T06:21:17.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 msgr2=0x7f0258155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f0258155fa0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f025000cd40 tx=0x7f025000a570 comp rx=0 tx=0).stop 2023-11-24T06:21:17.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 shutdown_connections 2023-11-24T06:21:17.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025815e080 0x7f0258155fa0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f024406aff0 0x7f024406d4b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f025806c710 0x7f025813eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 --2- 172.21.15.89:0/4252531303 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02580af4d0 0x7f025813f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 >> 172.21.15.89:0/4252531303 conn(0x7f0258071b90 msgr2=0x7f025806bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:17.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 shutdown_connections 2023-11-24T06:21:17.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.151+0000 7f025da38700 1 -- 172.21.15.89:0/4252531303 wait complete. 2023-11-24T06:21:17.166 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:18:37.761810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:20:55.237439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:20:55.790741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:21:17.166 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:21:17.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.287+0000 7fa728063700 1 Processor -- start 2023-11-24T06:21:17.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.287+0000 7fa728063700 1 -- start start 2023-11-24T06:21:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.288+0000 7fa728063700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa7200746a0 0x7fa7200b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.288+0000 7fa728063700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7200afc90 0x7fa72006d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.288+0000 7fa728063700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7200b3c60 con 0x7fa72015e160 2023-11-24T06:21:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.288+0000 7fa728063700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7200b4500 con 0x7fa7200afc90 2023-11-24T06:21:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.288+0000 7fa728063700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7200b58c0 con 0x7fa7200746a0 2023-11-24T06:21:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.288+0000 7fa726600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7200afc90 0x7fa72006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.288+0000 7fa725dff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa72015e160 0x7fa720073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42932/0 (socket says 172.21.15.89:42932) 2023-11-24T06:21:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa725dff700 1 -- 172.21.15.89:0/2066211643 learned_addr learned my addr 172.21.15.89:0/2066211643 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa7255fe700 1 --2- 172.21.15.89:0/2066211643 >> v2:172.21.15.123:3300/0 conn(0x7fa7200746a0 0x7fa7200b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa726600700 1 -- 172.21.15.89:0/2066211643 >> v2:172.21.15.123:3300/0 conn(0x7fa7200746a0 msgr2=0x7fa7200b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa726600700 1 --2- 172.21.15.89:0/2066211643 >> v2:172.21.15.123:3300/0 conn(0x7fa7200746a0 0x7fa7200b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa726600700 1 -- 172.21.15.89:0/2066211643 >> v1:172.21.15.89:6789/0 conn(0x7fa72015e160 legacy=0x7fa720073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa726600700 1 -- 172.21.15.89:0/2066211643 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72006bdc0 con 0x7fa7200afc90 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa724dfd700 1 -- 172.21.15.89:0/2066211643 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914935933 0 0) 0x7fa7200b3c60 con 0x7fa72015e160 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa726600700 1 --2- 172.21.15.89:0/2066211643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7200afc90 0x7fa72006d930 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa7200d0880 tx=0x7fa71800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c61de8fa949c46 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa724dfd700 1 -- 172.21.15.89:0/2066211643 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71800e070 con 0x7fa7200afc90 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa724dfd700 1 -- 172.21.15.89:0/2066211643 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71800bbd0 con 0x7fa7200afc90 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.289+0000 7fa724dfd700 1 -- 172.21.15.89:0/2066211643 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa718012430 con 0x7fa7200afc90 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.290+0000 7fa728063700 1 -- 172.21.15.89:0/2066211643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7200afc90 msgr2=0x7fa72006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.290+0000 7fa728063700 1 --2- 172.21.15.89:0/2066211643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7200afc90 0x7fa72006d930 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa7200d0880 tx=0x7fa71800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.290+0000 7fa728063700 1 -- 172.21.15.89:0/2066211643 shutdown_connections 2023-11-24T06:21:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.290+0000 7fa728063700 1 --2- 172.21.15.89:0/2066211643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7200afc90 0x7fa72006d930 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.290+0000 7fa728063700 1 --2- 172.21.15.89:0/2066211643 >> v2:172.21.15.123:3300/0 conn(0x7fa7200746a0 0x7fa7200b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.290+0000 7fa728063700 1 -- 172.21.15.89:0/2066211643 >> 172.21.15.89:0/2066211643 conn(0x7fa7200d0bf0 msgr2=0x7fa7200d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.290+0000 7fa728063700 1 -- 172.21.15.89:0/2066211643 shutdown_connections 2023-11-24T06:21:17.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.290+0000 7fa728063700 1 -- 172.21.15.89:0/2066211643 wait complete. 2023-11-24T06:21:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.291+0000 7fa728063700 1 Processor -- start 2023-11-24T06:21:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.291+0000 7fa728063700 1 -- start start 2023-11-24T06:21:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa728063700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7200746a0 0x7fa72006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa728063700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7200afc90 0x7fa720158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa728063700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72015e160 0x7fa720154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa728063700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa720158560 con 0x7fa72015e160 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa728063700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72015eff0 con 0x7fa7200746a0 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa728063700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa720158790 con 0x7fa7200afc90 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72015e160 0x7fa720154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72015e160 0x7fa720154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39438/0 (socket says 172.21.15.89:39438) 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 -- 172.21.15.89:0/2073026117 learned_addr learned my addr 172.21.15.89:0/2073026117 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa725dff700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7200746a0 0x7fa72006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa7255fe700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7200afc90 0x7fa720158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 -- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7200afc90 msgr2=0x7fa720158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7200afc90 0x7fa720158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 -- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7200746a0 msgr2=0x7fa72006ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7200746a0 0x7fa72006ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 -- 172.21.15.89:0/2073026117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa718009a70 con 0x7fa72015e160 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa7255fe700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7200afc90 0x7fa720158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa725dff700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7200746a0 0x7fa72006ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.292+0000 7fa726600700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72015e160 0x7fa720154960 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa720072ed0 tx=0x7fa718004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.293+0000 7fa7167fc700 1 -- 172.21.15.89:0/2073026117 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa718012430 con 0x7fa72015e160 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.293+0000 7fa7167fc700 1 -- 172.21.15.89:0/2073026117 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa718020070 con 0x7fa72015e160 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.293+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa720158a70 con 0x7fa72015e160 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.293+0000 7fa7167fc700 1 -- 172.21.15.89:0/2073026117 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa718004280 con 0x7fa72015e160 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.293+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72015f270 con 0x7fa72015e160 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.294+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72015f400 con 0x7fa72015e160 2023-11-24T06:21:17.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.294+0000 7fa7167fc700 1 -- 172.21.15.89:0/2073026117 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa7180043e0 con 0x7fa72015e160 2023-11-24T06:21:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.294+0000 7fa7167fc700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa70406aff0 0x7fa70406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.295+0000 7fa7167fc700 1 -- 172.21.15.89:0/2073026117 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa718022070 con 0x7fa72015e160 2023-11-24T06:21:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.295+0000 7fa725dff700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa70406aff0 0x7fa70406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.295+0000 7fa725dff700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa70406aff0 0x7fa70406d4b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa710000f60 tx=0x7fa710007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.299+0000 7fa7167fc700 1 -- 172.21.15.89:0/2073026117 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa718056540 con 0x7fa72015e160 2023-11-24T06:21:17.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.486+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa720068fb0 con 0x7fa72015e160 2023-11-24T06:21:17.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.486+0000 7fa7167fc700 1 -- 172.21.15.89:0/2073026117 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa718089e60 con 0x7fa72015e160 2023-11-24T06:21:17.489 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-11-24T06:21:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa70406aff0 msgr2=0x7fa70406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa70406aff0 0x7fa70406d4b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa710000f60 tx=0x7fa710007d30 comp rx=0 tx=0).stop 2023-11-24T06:21:17.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72015e160 msgr2=0x7fa720154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72015e160 0x7fa720154960 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa720072ed0 tx=0x7fa718004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 shutdown_connections 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72015e160 0x7fa720154960 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa70406aff0 0x7fa70406d4b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7200746a0 0x7fa72006ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 --2- 172.21.15.89:0/2073026117 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7200afc90 0x7fa720158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 >> 172.21.15.89:0/2073026117 conn(0x7fa7200d0bf0 msgr2=0x7fa72015d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 shutdown_connections 2023-11-24T06:21:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.488+0000 7fa728063700 1 -- 172.21.15.89:0/2073026117 wait complete. 2023-11-24T06:21:17.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:21:17.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.622+0000 7f3ddb708700 1 Processor -- start 2023-11-24T06:21:17.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.623+0000 7f3ddb708700 1 -- start start 2023-11-24T06:21:17.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.623+0000 7f3ddb708700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd40b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.623+0000 7f3ddb708700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3dd40afc90 0x7f3dd406d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.623+0000 7f3ddb708700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40b3c60 con 0x7f3dd415e160 2023-11-24T06:21:17.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.623+0000 7f3ddb708700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd40b4500 con 0x7f3dd40afc90 2023-11-24T06:21:17.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.623+0000 7f3ddb708700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b58c0 con 0x7f3dd40746a0 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd9ca5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3dd40afc90 0x7f3dd406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd9ca5700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3dd40afc90 0x7f3dd406d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39446/0 (socket says 172.21.15.89:39446) 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd94a4700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3dd415e160 0x7f3dd4073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42946/0 (socket says 172.21.15.89:42946) 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd9ca5700 1 -- 172.21.15.89:0/36207699 learned_addr learned my addr 172.21.15.89:0/36207699 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd8ca3700 1 --2- 172.21.15.89:0/36207699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd40b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd9ca5700 1 -- 172.21.15.89:0/36207699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 msgr2=0x7f3dd40b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd9ca5700 1 --2- 172.21.15.89:0/36207699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd40b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd9ca5700 1 -- 172.21.15.89:0/36207699 >> v1:172.21.15.89:6789/0 conn(0x7f3dd415e160 legacy=0x7f3dd4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dcbfff700 1 -- 172.21.15.89:0/36207699 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494363162 0 0) 0x7f3dd40b3c60 con 0x7f3dd415e160 2023-11-24T06:21:17.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd9ca5700 1 -- 172.21.15.89:0/36207699 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd40b0140 con 0x7f3dd40afc90 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd8ca3700 1 --2- 172.21.15.89:0/36207699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd40b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dd9ca5700 1 --2- 172.21.15.89:0/36207699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd40afc90 0x7f3dd406d930 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009e50 tx=0x7f3dcc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc320ad8823f250 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dcbfff700 1 -- 172.21.15.89:0/36207699 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc00e070 con 0x7f3dd40afc90 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dcbfff700 1 -- 172.21.15.89:0/36207699 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc00b9c0 con 0x7f3dd40afc90 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.624+0000 7f3dcbfff700 1 -- 172.21.15.89:0/36207699 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc00bd70 con 0x7f3dd40afc90 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.625+0000 7f3ddb708700 1 -- 172.21.15.89:0/36207699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd40afc90 msgr2=0x7f3dd406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.625+0000 7f3ddb708700 1 --2- 172.21.15.89:0/36207699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd40afc90 0x7f3dd406d930 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009e50 tx=0x7f3dcc00b330 comp rx=0 tx=0).stop 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.625+0000 7f3ddb708700 1 -- 172.21.15.89:0/36207699 shutdown_connections 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.625+0000 7f3ddb708700 1 --2- 172.21.15.89:0/36207699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd40afc90 0x7f3dd406d930 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.625+0000 7f3ddb708700 1 --2- 172.21.15.89:0/36207699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd40b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.625+0000 7f3ddb708700 1 -- 172.21.15.89:0/36207699 >> 172.21.15.89:0/36207699 conn(0x7f3dd40d0bf0 msgr2=0x7f3dd40d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:17.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.625+0000 7f3ddb708700 1 -- 172.21.15.89:0/36207699 shutdown_connections 2023-11-24T06:21:17.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.625+0000 7f3ddb708700 1 -- 172.21.15.89:0/36207699 wait complete. 2023-11-24T06:21:17.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.626+0000 7f3ddb708700 1 Processor -- start 2023-11-24T06:21:17.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.626+0000 7f3ddb708700 1 -- start start 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3ddb708700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd414faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3ddb708700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dd40afc90 0x7f3dd4154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3ddb708700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd415e160 0x7f3dd4152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3ddb708700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd41542a0 con 0x7f3dd415e160 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3ddb708700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd41534a0 con 0x7f3dd40afc90 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3ddb708700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd415ebd0 con 0x7f3dd40746a0 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd415e160 0x7f3dd4152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd415e160 0x7f3dd4152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39452/0 (socket says 172.21.15.89:39452) 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 -- 172.21.15.89:0/2820140953 learned_addr learned my addr 172.21.15.89:0/2820140953 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:17.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd8ca3700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dd40afc90 0x7f3dd4154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd94a4700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd414faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 -- 172.21.15.89:0/2820140953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 msgr2=0x7f3dd414faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd414faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 -- 172.21.15.89:0/2820140953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dd40afc90 msgr2=0x7f3dd4154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dd40afc90 0x7f3dd4154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 -- 172.21.15.89:0/2820140953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc009b30 con 0x7f3dd415e160 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dd9ca5700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd415e160 0x7f3dd4152f60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00b730 tx=0x7f3dcc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dc9ffb700 1 -- 172.21.15.89:0/2820140953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc0092e0 con 0x7f3dd415e160 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3dc9ffb700 1 -- 172.21.15.89:0/2820140953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc01c440 con 0x7f3dd415e160 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.627+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd4153720 con 0x7f3dd415e160 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.628+0000 7f3dc9ffb700 1 -- 172.21.15.89:0/2820140953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc01c810 con 0x7f3dd415e160 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.628+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd4153ce0 con 0x7f3dd415e160 2023-11-24T06:21:17.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.629+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd40b4c60 con 0x7f3dd415e160 2023-11-24T06:21:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.629+0000 7f3dc9ffb700 1 -- 172.21.15.89:0/2820140953 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3dcc01c970 con 0x7f3dd415e160 2023-11-24T06:21:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.629+0000 7f3dc9ffb700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3db806aff0 0x7f3db806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.629+0000 7f3dc9ffb700 1 -- 172.21.15.89:0/2820140953 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dcc08b300 con 0x7f3dd415e160 2023-11-24T06:21:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.630+0000 7f3dd94a4700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3db806aff0 0x7f3db806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.630+0000 7f3dd94a4700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3db806aff0 0x7f3db806d4b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4000f60 tx=0x7f3dc4007c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.634+0000 7f3dc9ffb700 1 -- 172.21.15.89:0/2820140953 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3dcc056220 con 0x7f3dd415e160 2023-11-24T06:21:17.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.823+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3dd415ed90 con 0x7f3dd415e160 2023-11-24T06:21:17.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.823+0000 7f3dc9ffb700 1 -- 172.21.15.89:0/2820140953 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v64) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3dcc059840 con 0x7f3dd415e160 2023-11-24T06:21:17.826 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-11-24T06:21:17.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3db806aff0 msgr2=0x7f3db806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3db806aff0 0x7f3db806d4b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4000f60 tx=0x7f3dc4007c90 comp rx=0 tx=0).stop 2023-11-24T06:21:17.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd415e160 msgr2=0x7f3dd4152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd415e160 0x7f3dd4152f60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00b730 tx=0x7f3dcc004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 shutdown_connections 2023-11-24T06:21:17.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3dd415e160 0x7f3dd4152f60 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dd40746a0 0x7f3dd414faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3db806aff0 0x7f3db806d4b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 --2- 172.21.15.89:0/2820140953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dd40afc90 0x7f3dd4154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 >> 172.21.15.89:0/2820140953 conn(0x7f3dd40d0bf0 msgr2=0x7f3dd41623f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:17.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 shutdown_connections 2023-11-24T06:21:17.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.825+0000 7f3ddb708700 1 -- 172.21.15.89:0/2820140953 wait complete. 2023-11-24T06:21:17.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:21:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.961+0000 7fa175fa8700 1 Processor -- start 2023-11-24T06:21:17.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.961+0000 7fa175fa8700 1 -- start start 2023-11-24T06:21:17.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.961+0000 7fa175fa8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1700b5da0 0x7fa1700afc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.961+0000 7fa175fa8700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1700b3c60 con 0x7fa1700746a0 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.961+0000 7fa175fa8700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1700b58c0 con 0x7fa17015e160 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.961+0000 7fa175fa8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700b4500 con 0x7fa1700b5da0 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16effd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa1700746a0 0x7fa17006d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42962/0 (socket says 172.21.15.89:42962) 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16effd700 1 -- 172.21.15.89:0/2605198931 learned_addr learned my addr 172.21.15.89:0/2605198931 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16ffff700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1700b5da0 0x7fa1700afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844155200 0 0) 0x7fa1700b3c60 con 0x7fa1700746a0 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c0035b0 con 0x7fa1700746a0 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580543947 0 0) 0x7fa15c0035b0 con 0x7fa1700746a0 2023-11-24T06:21:17.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1700b5da0 msgr2=0x7fa1700afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:17.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1700b5da0 0x7fa1700afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 >> v1:172.21.15.123:6789/0 conn(0x7fa17015e160 legacy=0x7fa170073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17006bdc0 con 0x7fa1700746a0 2023-11-24T06:21:17.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fa158001670 con 0x7fa1700746a0 2023-11-24T06:21:17.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 >> v1:172.21.15.89:6789/0 conn(0x7fa1700746a0 legacy=0x7fa17006d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15c003a30 0x7fa15c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa15c0043e0 0x7fa15c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1700b5da0 0x7fa15c005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700b4500 con 0x7fa15c003a30 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c00ba10 con 0x7fa15c0043e0 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c00bbd0 con 0x7fa1700b5da0 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16f7fe700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15c003a30 0x7fa15c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16effd700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa15c0043e0 0x7fa15c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16ffff700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1700b5da0 0x7fa15c005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16f7fe700 1 -- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1700b5da0 msgr2=0x7fa15c005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16f7fe700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1700b5da0 0x7fa15c005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16f7fe700 1 -- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa15c0043e0 msgr2=0x7fa15c0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16f7fe700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa15c0043e0 0x7fa15c0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.962+0000 7fa16f7fe700 1 -- 172.21.15.89:0/2605198931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c0035b0 con 0x7fa15c003a30 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa16f7fe700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15c003a30 0x7fa15c003e10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa1600018b0 tx=0x7fa15c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7508643f9fbb3925 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160001370 con 0x7fa15c003a30 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16000c6b0 con 0x7fa15c003a30 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa16e7fc700 1 -- 172.21.15.89:0/2605198931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16000ca60 con 0x7fa15c003a30 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa175fa8700 1 -- 172.21.15.89:0/2605198931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15c003a30 msgr2=0x7fa15c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa175fa8700 1 --2- 172.21.15.89:0/2605198931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15c003a30 0x7fa15c003e10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa1600018b0 tx=0x7fa15c005880 comp rx=0 tx=0).stop 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa16ffff700 1 -- 172.21.15.89:0/2605198931 reap_dead start 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa175fa8700 1 -- 172.21.15.89:0/2605198931 shutdown_connections 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa175fa8700 1 -- 172.21.15.89:0/2605198931 >> 172.21.15.89:0/2605198931 conn(0x7fa1700d0bf0 msgr2=0x7fa1700d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:17.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa175fa8700 1 -- 172.21.15.89:0/2605198931 shutdown_connections 2023-11-24T06:21:17.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.963+0000 7fa175fa8700 1 -- 172.21.15.89:0/2605198931 wait complete. 2023-11-24T06:21:17.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.964+0000 7fa175fa8700 1 Processor -- start 2023-11-24T06:21:17.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa175fa8700 1 -- start start 2023-11-24T06:21:17.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa175fa8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa15c003a30 0x7fa170155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa175fa8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1700cfe80 0x7fa17014e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa175fa8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa170157710 0x7fa170157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa175fa8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa170156d10 con 0x7fa170157710 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa175fa8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1701580d0 con 0x7fa1700cfe80 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa175fa8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa170156190 con 0x7fa15c003a30 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa170157710 0x7fa170157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.965+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa170157710 0x7fa170157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39472/0 (socket says 172.21.15.89:39472) 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16ffff700 1 -- 172.21.15.89:0/2280491547 learned_addr learned my addr 172.21.15.89:0/2280491547 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16effd700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1700cfe80 0x7fa17014e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16f7fe700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa15c003a30 0x7fa170155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16ffff700 1 -- 172.21.15.89:0/2280491547 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa15c003a30 msgr2=0x7fa170155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16ffff700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa15c003a30 0x7fa170155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16ffff700 1 -- 172.21.15.89:0/2280491547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1700cfe80 msgr2=0x7fa17014e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:17.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16ffff700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1700cfe80 0x7fa17014e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:17.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16ffff700 1 -- 172.21.15.89:0/2280491547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa170156470 con 0x7fa170157710 2023-11-24T06:21:17.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa16ffff700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa170157710 0x7fa170157b90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa1700b5070 tx=0x7fa1680114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa167fff700 1 -- 172.21.15.89:0/2280491547 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16800e070 con 0x7fa170157710 2023-11-24T06:21:17.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa167fff700 1 -- 172.21.15.89:0/2280491547 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168003cf0 con 0x7fa170157710 2023-11-24T06:21:17.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa167fff700 1 -- 172.21.15.89:0/2280491547 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168003ea0 con 0x7fa170157710 2023-11-24T06:21:17.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa170156750 con 0x7fa170157710 2023-11-24T06:21:17.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.966+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa170161520 con 0x7fa170157710 2023-11-24T06:21:17.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.967+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa170068fb0 con 0x7fa170157710 2023-11-24T06:21:17.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.968+0000 7fa167fff700 1 -- 172.21.15.89:0/2280491547 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa168011960 con 0x7fa170157710 2023-11-24T06:21:17.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.968+0000 7fa167fff700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa15006aff0 0x7fa15006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:17.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.968+0000 7fa167fff700 1 -- 172.21.15.89:0/2280491547 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa168089b00 con 0x7fa170157710 2023-11-24T06:21:17.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.968+0000 7fa16f7fe700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa15006aff0 0x7fa15006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:17.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.968+0000 7fa16f7fe700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa15006aff0 0x7fa15006d4b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa160014040 tx=0x7fa160005040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:17.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:17.972+0000 7fa167fff700 1 -- 172.21.15.89:0/2280491547 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa168054990 con 0x7fa170157710 2023-11-24T06:21:18.116 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:21:18.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:21:18.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.170+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1700af4b0 con 0x7fa170157710 2023-11-24T06:21:18.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.170+0000 7fa167fff700 1 -- 172.21.15.89:0/2280491547 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v64) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa168057fb0 con 0x7fa170157710 2023-11-24T06:21:18.173 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-11-24T06:21:18.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa15006aff0 msgr2=0x7fa15006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa15006aff0 0x7fa15006d4b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa160014040 tx=0x7fa160005040 comp rx=0 tx=0).stop 2023-11-24T06:21:18.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa170157710 msgr2=0x7fa170157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa170157710 0x7fa170157b90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa1700b5070 tx=0x7fa1680114b0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 shutdown_connections 2023-11-24T06:21:18.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa170157710 0x7fa170157b90 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa15c003a30 0x7fa170155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa15006aff0 0x7fa15006d4b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 --2- 172.21.15.89:0/2280491547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1700cfe80 0x7fa17014e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 >> 172.21.15.89:0/2280491547 conn(0x7fa1700d0bf0 msgr2=0x7fa170162510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:18.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 shutdown_connections 2023-11-24T06:21:18.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.172+0000 7fa175fa8700 1 -- 172.21.15.89:0/2280491547 wait complete. 2023-11-24T06:21:18.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:21:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.343+0000 7f5403fff700 1 Processor -- start 2023-11-24T06:21:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.343+0000 7f5403fff700 1 -- start start 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.344+0000 7f5403fff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f540415ba80 0x7f54040cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.344+0000 7f5403fff700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54040b1d20 con 0x7f540415be60 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.344+0000 7f5403fff700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54040cf560 con 0x7f54040ce360 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.344+0000 7f5403fff700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54040b1f80 con 0x7f540415ba80 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.344+0000 7f54037fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f540415be60 0x7f54040d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42984/0 (socket says 172.21.15.89:42984) 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.344+0000 7f54037fe700 1 -- 172.21.15.89:0/669483255 learned_addr learned my addr 172.21.15.89:0/669483255 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5402ffd700 1 --2- 172.21.15.89:0/669483255 >> v2:172.21.15.123:3300/0 conn(0x7f540415ba80 0x7f54040cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690661799 0 0) 0x7f54040b1d20 con 0x7f540415be60 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ec0035b0 con 0x7f540415be60 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106058067 0 0) 0x7f53ec0035b0 con 0x7f540415be60 2023-11-24T06:21:18.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 >> v2:172.21.15.123:3300/0 conn(0x7f540415ba80 msgr2=0x7f54040cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:18.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 --2- 172.21.15.89:0/669483255 >> v2:172.21.15.123:3300/0 conn(0x7f540415ba80 0x7f54040cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 >> v1:172.21.15.123:6789/0 conn(0x7f54040ce360 legacy=0x7f54040b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54040b2260 con 0x7f540415be60 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f53fc0019f0 con 0x7f540415be60 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 >> v1:172.21.15.89:6789/0 conn(0x7f540415be60 legacy=0x7f54040d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004850 0x7f53ec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec005210 0x7f53ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54040b1f80 con 0x7f53ec004850 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec00c1b0 con 0x7f53ec003a00 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec00c370 con 0x7f53ec005210 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54027fc700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004850 0x7f53ec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54037fe700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec005210 0x7f53ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f5402ffd700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54027fc700 1 -- 172.21.15.89:0/669483255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec005210 msgr2=0x7f53ec00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54027fc700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec005210 0x7f53ec00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54027fc700 1 -- 172.21.15.89:0/669483255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec003a00 msgr2=0x7f53ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54027fc700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec003a00 0x7f53ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54027fc700 1 -- 172.21.15.89:0/669483255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0035b0 con 0x7f53ec004850 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54037fe700 1 -- 172.21.15.89:0/669483255 reap_dead start 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.345+0000 7f54027fc700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004850 0x7f53ec004cd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f53f40017c0 tx=0x7f53ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d43386f3b860b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40052d0 con 0x7f53ec004850 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f4005450 con 0x7f53ec004850 2023-11-24T06:21:18.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5401ffb700 1 -- 172.21.15.89:0/669483255 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40038c0 con 0x7f53ec004850 2023-11-24T06:21:18.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5403fff700 1 -- 172.21.15.89:0/669483255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004850 msgr2=0x7f53ec004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5403fff700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004850 0x7f53ec004cd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f53f40017c0 tx=0x7f53ec005880 comp rx=0 tx=0).stop 2023-11-24T06:21:18.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5403fff700 1 -- 172.21.15.89:0/669483255 shutdown_connections 2023-11-24T06:21:18.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5403fff700 1 --2- 172.21.15.89:0/669483255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53ec004850 0x7f53ec004cd0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5403fff700 1 -- 172.21.15.89:0/669483255 >> 172.21.15.89:0/669483255 conn(0x7f540409aa00 msgr2=0x7f540409ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:18.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5403fff700 1 -- 172.21.15.89:0/669483255 shutdown_connections 2023-11-24T06:21:18.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.346+0000 7f5403fff700 1 -- 172.21.15.89:0/669483255 wait complete. 2023-11-24T06:21:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.347+0000 7f5403fff700 1 Processor -- start 2023-11-24T06:21:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f5403fff700 1 -- start start 2023-11-24T06:21:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f5403fff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004850 0x7f54040b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f5403fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f540415ecc0 0x7f54040ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f5403fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54041f6f80 0x7f54041f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f5403fff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540415c270 con 0x7f540415ecc0 2023-11-24T06:21:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f5403fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54041f7950 con 0x7f54041f6f80 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f5403fff700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5404151900 con 0x7f53ec004850 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f54027fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f540415ecc0 0x7f54040ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f54027fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f540415ecc0 0x7f54040ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39490/0 (socket says 172.21.15.89:39490) 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f54027fc700 1 -- 172.21.15.89:0/2334838600 learned_addr learned my addr 172.21.15.89:0/2334838600 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f5402ffd700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004850 0x7f54040b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.348+0000 7f54037fe700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54041f6f80 0x7f54041f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f54027fc700 1 -- 172.21.15.89:0/2334838600 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004850 msgr2=0x7f54040b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f54027fc700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004850 0x7f54040b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f54027fc700 1 -- 172.21.15.89:0/2334838600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54041f6f80 msgr2=0x7f54041f7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f54027fc700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54041f6f80 0x7f54041f7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f54027fc700 1 -- 172.21.15.89:0/2334838600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540406aff0 con 0x7f540415ecc0 2023-11-24T06:21:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f54027fc700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f540415ecc0 0x7f54040ca3f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00c180 tx=0x7f53f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f53eb7fe700 1 -- 172.21.15.89:0/2334838600 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4001290 con 0x7f540415ecc0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f53eb7fe700 1 -- 172.21.15.89:0/2334838600 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f4007040 con 0x7f540415ecc0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54040ca930 con 0x7f540415ecc0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f53eb7fe700 1 -- 172.21.15.89:0/2334838600 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40073f0 con 0x7f540415ecc0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.349+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54040caeb0 con 0x7f540415ecc0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.350+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5404074360 con 0x7f540415ecc0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.350+0000 7f53eb7fe700 1 -- 172.21.15.89:0/2334838600 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f53f4007550 con 0x7f540415ecc0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.351+0000 7f53eb7fe700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53f006aff0 0x7f53f006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.351+0000 7f53eb7fe700 1 -- 172.21.15.89:0/2334838600 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53f408a0f0 con 0x7f540415ecc0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.351+0000 7f5402ffd700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53f006aff0 0x7f53f006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.351+0000 7f5402ffd700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53f006aff0 0x7f53f006d4b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f540406cbc0 tx=0x7f53f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.355+0000 7f53eb7fe700 1 -- 172.21.15.89:0/2334838600 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53f40552f0 con 0x7f540415ecc0 2023-11-24T06:21:18.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.586+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54040751e0 con 0x7f540415ecc0 2023-11-24T06:21:18.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.586+0000 7f53eb7fe700 1 -- 172.21.15.89:0/2334838600 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f53f4058910 con 0x7f540415ecc0 2023-11-24T06:21:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:18.590 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/351443905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":351443905},{"type":"v1","addr":"172.21.15.123:6834","nonce":351443905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6021,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2890850053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2890850053},{"type":"v1","addr":"172.21.15.89:6836","nonce":2890850053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3912248946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3912248946},{"type":"v1","addr":"172.21.15.89:6837","nonce":3912248946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:56.834302+0000","modified":"2023-11-24T06:20:59.846940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6039":{"gid":6039,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2963752713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2963752713},{"type":"v1","addr":"172.21.15.123:6835","nonce":2963752713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T06:21:18.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53f006aff0 msgr2=0x7f53f006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53f006aff0 0x7f53f006d4b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f540406cbc0 tx=0x7f53f8008040 comp rx=0 tx=0).stop 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f540415ecc0 msgr2=0x7f54040ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f540415ecc0 0x7f54040ca3f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00c180 tx=0x7f53f4001120 comp rx=0 tx=0).stop 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 shutdown_connections 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f53f006aff0 0x7f53f006d4b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54041f6f80 0x7f54041f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53ec004850 0x7f54040b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 --2- 172.21.15.89:0/2334838600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f540415ecc0 0x7f54040ca3f0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.588+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 >> 172.21.15.89:0/2334838600 conn(0x7f540409aa00 msgr2=0x7f54040d21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.589+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 shutdown_connections 2023-11-24T06:21:18.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.589+0000 7f5403fff700 1 -- 172.21.15.89:0/2334838600 wait complete. 2023-11-24T06:21:18.593 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 42 2023-11-24T06:21:18.604 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:21:18.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.720+0000 7fe53048b700 1 Processor -- start 2023-11-24T06:21:18.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.720+0000 7fe53048b700 1 -- start start 2023-11-24T06:21:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.721+0000 7fe53048b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52815c160 0x7fe5280622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.721+0000 7fe53048b700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52815c540 con 0x7fe5280b2870 2023-11-24T06:21:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.721+0000 7fe53048b700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280b2ee0 con 0x7fe5280b04b0 2023-11-24T06:21:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.721+0000 7fe53048b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280b2cf0 con 0x7fe52815c160 2023-11-24T06:21:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.721+0000 7fe52da26700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe5280b2870 0x7fe5280afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43000/0 (socket says 172.21.15.89:43000) 2023-11-24T06:21:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.721+0000 7fe52da26700 1 -- 172.21.15.89:0/1046524696 learned_addr learned my addr 172.21.15.89:0/1046524696 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52e227700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52815c160 0x7fe5280622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069663140 0 0) 0x7fe52815c540 con 0x7fe5280b2870 2023-11-24T06:21:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5140035b0 con 0x7fe5280b2870 2023-11-24T06:21:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599987352 0 0) 0x7fe520001490 con 0x7fe5280b04b0 2023-11-24T06:21:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe52815c540 con 0x7fe5280b04b0 2023-11-24T06:21:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 355992061 0 0) 0x7fe5140035b0 con 0x7fe5280b2870 2023-11-24T06:21:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52815c160 msgr2=0x7fe5280622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52815c160 0x7fe5280622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 >> v1:172.21.15.123:6789/0 conn(0x7fe5280b04b0 legacy=0x7fe528165c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280add40 con 0x7fe5280b2870 2023-11-24T06:21:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7fe524001670 con 0x7fe5280b2870 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 >> v1:172.21.15.89:6789/0 conn(0x7fe5280b2870 legacy=0x7fe5280afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe514003bc0 0x7fe514003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe514004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52815c160 0x7fe514005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51400bac0 con 0x7fe514004850 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51400bc00 con 0x7fe514003bc0 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51400bdc0 con 0x7fe52815c160 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52da26700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe514004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52e227700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe514003bc0 0x7fe514003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52ea28700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52815c160 0x7fe514005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52da26700 1 -- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52815c160 msgr2=0x7fe514005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52da26700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52815c160 0x7fe514005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52da26700 1 -- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe514003bc0 msgr2=0x7fe514003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52da26700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe514003bc0 0x7fe514003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.722+0000 7fe52da26700 1 -- 172.21.15.89:0/1046524696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5140035b0 con 0x7fe514004850 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe52da26700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe514004cd0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe52400c160 tx=0x7fe514005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6f6c0d9c838697 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52400cc40 con 0x7fe514004850 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52400cda0 con 0x7fe514004850 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe52d225700 1 -- 172.21.15.89:0/1046524696 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524005ba0 con 0x7fe514004850 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe53048b700 1 -- 172.21.15.89:0/1046524696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 msgr2=0x7fe514004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe53048b700 1 --2- 172.21.15.89:0/1046524696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe514004cd0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe52400c160 tx=0x7fe514005880 comp rx=0 tx=0).stop 2023-11-24T06:21:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe52ea28700 1 -- 172.21.15.89:0/1046524696 reap_dead start 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe53048b700 1 -- 172.21.15.89:0/1046524696 shutdown_connections 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe53048b700 1 -- 172.21.15.89:0/1046524696 >> 172.21.15.89:0/1046524696 conn(0x7fe5280b57b0 msgr2=0x7fe5280b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe53048b700 1 -- 172.21.15.89:0/1046524696 shutdown_connections 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.723+0000 7fe53048b700 1 -- 172.21.15.89:0/1046524696 wait complete. 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe53048b700 1 Processor -- start 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe53048b700 1 -- start start 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe53048b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe52814d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe53048b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5280ccdf0 0x7fe5280cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe53048b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52814f4e0 0x7fe52814f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe53048b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280cd7b0 con 0x7fe514004850 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe53048b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52815c6d0 con 0x7fe5280ccdf0 2023-11-24T06:21:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe53048b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52815d530 con 0x7fe52814f4e0 2023-11-24T06:21:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe52e227700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe52814d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe52e227700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe52814d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39506/0 (socket says 172.21.15.89:39506) 2023-11-24T06:21:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.725+0000 7fe52e227700 1 -- 172.21.15.89:0/3805468518 learned_addr learned my addr 172.21.15.89:0/3805468518 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52da26700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5280ccdf0 0x7fe5280cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52ea28700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52814f4e0 0x7fe52814f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52e227700 1 -- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52814f4e0 msgr2=0x7fe52814f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52e227700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52814f4e0 0x7fe52814f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52e227700 1 -- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5280ccdf0 msgr2=0x7fe5280cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52e227700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5280ccdf0 0x7fe5280cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52e227700 1 -- 172.21.15.89:0/3805468518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52814d800 con 0x7fe514004850 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52da26700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5280ccdf0 0x7fe5280cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe52e227700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe52814d2c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe5180119d0 tx=0x7fe518011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe51effd700 1 -- 172.21.15.89:0/3805468518 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51800f8e0 con 0x7fe514004850 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe51effd700 1 -- 172.21.15.89:0/3805468518 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51800fa40 con 0x7fe514004850 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52814fea0 con 0x7fe514004850 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe51effd700 1 -- 172.21.15.89:0/3805468518 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51800fdd0 con 0x7fe514004850 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.726+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52815cc00 con 0x7fe514004850 2023-11-24T06:21:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.727+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5280ad6b0 con 0x7fe514004850 2023-11-24T06:21:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.727+0000 7fe51effd700 1 -- 172.21.15.89:0/3805468518 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe518014440 con 0x7fe514004850 2023-11-24T06:21:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.728+0000 7fe51effd700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe50c06aff0 0x7fe50c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.728+0000 7fe52da26700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe50c06aff0 0x7fe50c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.728+0000 7fe51effd700 1 -- 172.21.15.89:0/3805468518 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe518089dc0 con 0x7fe514004850 2023-11-24T06:21:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.728+0000 7fe52da26700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe50c06aff0 0x7fe50c06d4b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe524015040 tx=0x7fe524007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.732+0000 7fe51effd700 1 -- 172.21.15.89:0/3805468518 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe518054d60 con 0x7fe514004850 2023-11-24T06:21:18.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.924+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5280b1b50 con 0x7fe514004850 2023-11-24T06:21:18.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.925+0000 7fe51effd700 1 -- 172.21.15.89:0/3805468518 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16532 (secure 0 0 0) 0x7fe518058380 con 0x7fe514004850 2023-11-24T06:21:18.929 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:18.929 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":64,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:58.434050+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:55.237439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:55.790741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:21:18.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe50c06aff0 msgr2=0x7fe50c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe50c06aff0 0x7fe50c06d4b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe524015040 tx=0x7fe524007000 comp rx=0 tx=0).stop 2023-11-24T06:21:18.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 msgr2=0x7fe52814d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe52814d2c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe5180119d0 tx=0x7fe518011d90 comp rx=0 tx=0).stop 2023-11-24T06:21:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 shutdown_connections 2023-11-24T06:21:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe52814f4e0 0x7fe52814f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe514004850 0x7fe52814d2c0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe50c06aff0 0x7fe50c06d4b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 --2- 172.21.15.89:0/3805468518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5280ccdf0 0x7fe5280cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 >> 172.21.15.89:0/3805468518 conn(0x7fe5280b57b0 msgr2=0x7fe5280726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 shutdown_connections 2023-11-24T06:21:18.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:18.927+0000 7fe53048b700 1 -- 172.21.15.89:0/3805468518 wait complete. 2023-11-24T06:21:18.943 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T06:21:18.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:21:19.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.062+0000 7f7f2f623700 1 Processor -- start 2023-11-24T06:21:19.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.062+0000 7f7f2f623700 1 -- start start 2023-11-24T06:21:19.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2f623700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7f281446a0 0x7f7f280b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2f623700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7f280adf00 0x7f7f2806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2f623700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 0x7f7f2806da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2f623700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f28144a80 con 0x7f7f281446a0 2023-11-24T06:21:19.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2f623700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2806c9c0 con 0x7f7f280adf00 2023-11-24T06:21:19.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2f623700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f280af950 con 0x7f7f2806c540 2023-11-24T06:21:19.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2d3bf700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7f281446a0 0x7f7f280b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2d3bf700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7f281446a0 0x7f7f280b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39514/0 (socket says 172.21.15.89:39514) 2023-11-24T06:21:19.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.063+0000 7f7f2d3bf700 1 -- 172.21.15.89:0/3218648704 learned_addr learned my addr 172.21.15.89:0/3218648704 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2d3bf700 1 -- 172.21.15.89:0/3218648704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 msgr2=0x7f7f2806da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2d3bf700 1 --2- 172.21.15.89:0/3218648704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 0x7f7f2806da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2d3bf700 1 -- 172.21.15.89:0/3218648704 >> v2:172.21.15.123:3300/0 conn(0x7f7f280adf00 msgr2=0x7f7f2806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2cbbe700 1 --2- 172.21.15.89:0/3218648704 >> v2:172.21.15.123:3300/0 conn(0x7f7f280adf00 0x7f7f2806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2d3bf700 1 --2- 172.21.15.89:0/3218648704 >> v2:172.21.15.123:3300/0 conn(0x7f7f280adf00 0x7f7f2806c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2d3bf700 1 -- 172.21.15.89:0/3218648704 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280afc30 con 0x7f7f281446a0 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2d3bf700 1 --2- 172.21.15.89:0/3218648704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f280b5ae0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7f2809f030 tx=0x7f7f18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f48765b1f95e17 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f1ffff700 1 -- 172.21.15.89:0/3218648704 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1800b870 con 0x7f7f281446a0 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f1ffff700 1 -- 172.21.15.89:0/3218648704 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1800b9d0 con 0x7f7f281446a0 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f1ffff700 1 -- 172.21.15.89:0/3218648704 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1800bd60 con 0x7f7f281446a0 2023-11-24T06:21:19.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2f623700 1 -- 172.21.15.89:0/3218648704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 msgr2=0x7f7f280b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.064+0000 7f7f2f623700 1 --2- 172.21.15.89:0/3218648704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f280b5ae0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7f2809f030 tx=0x7f7f18009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.065+0000 7f7f2f623700 1 -- 172.21.15.89:0/3218648704 shutdown_connections 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.065+0000 7f7f2f623700 1 --2- 172.21.15.89:0/3218648704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 0x7f7f2806da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.065+0000 7f7f2f623700 1 --2- 172.21.15.89:0/3218648704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f280b5ae0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.065+0000 7f7f2f623700 1 --2- 172.21.15.89:0/3218648704 >> v2:172.21.15.123:3300/0 conn(0x7f7f280adf00 0x7f7f2806c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.065+0000 7f7f2f623700 1 -- 172.21.15.89:0/3218648704 >> 172.21.15.89:0/3218648704 conn(0x7f7f28071b90 msgr2=0x7f7f28071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.065+0000 7f7f2f623700 1 -- 172.21.15.89:0/3218648704 shutdown_connections 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.065+0000 7f7f2f623700 1 -- 172.21.15.89:0/3218648704 wait complete. 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.066+0000 7f7f2f623700 1 Processor -- start 2023-11-24T06:21:19.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.066+0000 7f7f2f623700 1 -- start start 2023-11-24T06:21:19.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2f623700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 0x7f7f28140330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2f623700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f280adf00 0x7f7f28140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2f623700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f2813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2f623700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2813af40 con 0x7f7f281446a0 2023-11-24T06:21:19.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2f623700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28141060 con 0x7f7f280adf00 2023-11-24T06:21:19.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2f623700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28141340 con 0x7f7f2806c540 2023-11-24T06:21:19.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f2813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f2813aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39524/0 (socket says 172.21.15.89:39524) 2023-11-24T06:21:19.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 -- 172.21.15.89:0/1518029549 learned_addr learned my addr 172.21.15.89:0/1518029549 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:19.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2d3bf700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 0x7f7f28140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2cbbe700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f280adf00 0x7f7f28140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 -- 172.21.15.89:0/1518029549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 msgr2=0x7f7f28140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 0x7f7f28140330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 -- 172.21.15.89:0/1518029549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f280adf00 msgr2=0x7f7f28140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f280adf00 0x7f7f28140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 -- 172.21.15.89:0/1518029549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f18009550 con 0x7f7f281446a0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2dbc0700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f2813aa00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000b970 tx=0x7f7f2000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f1dffb700 1 -- 172.21.15.89:0/1518029549 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2000c9b0 con 0x7f7f281446a0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f1dffb700 1 -- 172.21.15.89:0/1518029549 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2000cb30 con 0x7f7f281446a0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f1dffb700 1 -- 172.21.15.89:0/1518029549 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f20014440 con 0x7f7f281446a0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.067+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f28141620 con 0x7f7f281446a0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.068+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f28141be0 con 0x7f7f281446a0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.068+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f280753f0 con 0x7f7f281446a0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.069+0000 7f7f1dffb700 1 -- 172.21.15.89:0/1518029549 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f7f2001d3b0 con 0x7f7f281446a0 2023-11-24T06:21:19.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.069+0000 7f7f1dffb700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f0c06aff0 0x7f7f0c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.069+0000 7f7f1dffb700 1 -- 172.21.15.89:0/1518029549 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f20006e40 con 0x7f7f281446a0 2023-11-24T06:21:19.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.069+0000 7f7f2d3bf700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f0c06aff0 0x7f7f0c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.070+0000 7f7f2d3bf700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f0c06aff0 0x7f7f0c06d4b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7f18009c10 tx=0x7f7f18008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.074+0000 7f7f1dffb700 1 -- 172.21.15.89:0/1518029549 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f200544e0 con 0x7f7f281446a0 2023-11-24T06:21:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.309+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f280ae2e0 con 0x7f7f281446a0 2023-11-24T06:21:19.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.310+0000 7f7f1dffb700 1 -- 172.21.15.89:0/1518029549 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f7f20057b00 con 0x7f7f281446a0 2023-11-24T06:21:19.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:19.313 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/351443905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":351443905},{"type":"v1","addr":"172.21.15.123:6834","nonce":351443905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6021,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2890850053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2890850053},{"type":"v1","addr":"172.21.15.89:6836","nonce":2890850053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3912248946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3912248946},{"type":"v1","addr":"172.21.15.89:6837","nonce":3912248946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:56.834302+0000","modified":"2023-11-24T06:20:59.846940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6039":{"gid":6039,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2963752713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2963752713},{"type":"v1","addr":"172.21.15.123:6835","nonce":2963752713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T06:21:19.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f0c06aff0 msgr2=0x7f7f0c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f0c06aff0 0x7f7f0c06d4b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7f18009c10 tx=0x7f7f18008030 comp rx=0 tx=0).stop 2023-11-24T06:21:19.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 msgr2=0x7f7f2813aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f2813aa00 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000b970 tx=0x7f7f2000bd30 comp rx=0 tx=0).stop 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 shutdown_connections 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f281446a0 0x7f7f2813aa00 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f2806c540 0x7f7f28140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f7f0c06aff0 0x7f7f0c06d4b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 --2- 172.21.15.89:0/1518029549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f280adf00 0x7f7f28140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 >> 172.21.15.89:0/1518029549 conn(0x7f7f28071b90 msgr2=0x7f7f2815d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 shutdown_connections 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.312+0000 7f7f2f623700 1 -- 172.21.15.89:0/1518029549 wait complete. 2023-11-24T06:21:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 42 2023-11-24T06:21:19.327 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:21:19.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.444+0000 7f0d50272700 1 Processor -- start 2023-11-24T06:21:19.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.444+0000 7f0d50272700 1 -- start start 2023-11-24T06:21:19.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d50272700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d480b3370 0x7f0d481604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d50272700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d480d1c70 0x7f0d480d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d50272700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d480b22d0 con 0x7f0d4815bdd0 2023-11-24T06:21:19.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d50272700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d480b2f50 con 0x7f0d480b3370 2023-11-24T06:21:19.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d50272700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480d2640 con 0x7f0d480d1c70 2023-11-24T06:21:19.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d80d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d480b3370 0x7f0d481604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d80d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0d480b3370 0x7f0d481604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39530/0 (socket says 172.21.15.89:39530) 2023-11-24T06:21:19.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d80d700 1 -- 172.21.15.89:0/3176421730 learned_addr learned my addr 172.21.15.89:0/3176421730 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d80d700 1 -- 172.21.15.89:0/3176421730 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d480d1c70 msgr2=0x7f0d480d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d80d700 1 --2- 172.21.15.89:0/3176421730 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d480d1c70 0x7f0d480d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d80d700 1 -- 172.21.15.89:0/3176421730 >> v1:172.21.15.123:6789/0 conn(0x7f0d4815bdd0 legacy=0x7f0d480b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d80d700 1 -- 172.21.15.89:0/3176421730 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d480b25b0 con 0x7f0d480b3370 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d80d700 1 --2- 172.21.15.89:0/3176421730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d481604e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0d380098f0 tx=0x7f0d38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2b39e0c31f49c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d00c700 1 -- 172.21.15.89:0/3176421730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d38004230 con 0x7f0d480b3370 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d00c700 1 -- 172.21.15.89:0/3176421730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d38004390 con 0x7f0d480b3370 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.445+0000 7f0d4d00c700 1 -- 172.21.15.89:0/3176421730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d380036a0 con 0x7f0d480b3370 2023-11-24T06:21:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.446+0000 7f0d50272700 1 -- 172.21.15.89:0/3176421730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 msgr2=0x7f0d481604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.446+0000 7f0d50272700 1 --2- 172.21.15.89:0/3176421730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d481604e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0d380098f0 tx=0x7f0d38009c00 comp rx=0 tx=0).stop 2023-11-24T06:21:19.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.446+0000 7f0d50272700 1 -- 172.21.15.89:0/3176421730 shutdown_connections 2023-11-24T06:21:19.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.446+0000 7f0d50272700 1 --2- 172.21.15.89:0/3176421730 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d480d1c70 0x7f0d480d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.446+0000 7f0d50272700 1 --2- 172.21.15.89:0/3176421730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d481604e0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.446+0000 7f0d50272700 1 -- 172.21.15.89:0/3176421730 >> 172.21.15.89:0/3176421730 conn(0x7f0d480731a0 msgr2=0x7f0d4809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:19.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.446+0000 7f0d50272700 1 -- 172.21.15.89:0/3176421730 shutdown_connections 2023-11-24T06:21:19.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.446+0000 7f0d50272700 1 -- 172.21.15.89:0/3176421730 wait complete. 2023-11-24T06:21:19.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.447+0000 7f0d50272700 1 Processor -- start 2023-11-24T06:21:19.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.447+0000 7f0d50272700 1 -- start start 2023-11-24T06:21:19.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d50272700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d480cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d50272700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d480d1c70 0x7f0d480c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d50272700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d4815bdd0 0x7f0d481537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d50272700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480cb1e0 con 0x7f0d480b3370 2023-11-24T06:21:19.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d50272700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480cbbb0 con 0x7f0d480d1c70 2023-11-24T06:21:19.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d50272700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d480ca5d0 con 0x7f0d4815bdd0 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e00e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d480cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e00e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d480cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39536/0 (socket says 172.21.15.89:39536) 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e00e700 1 -- 172.21.15.89:0/3081032560 learned_addr learned my addr 172.21.15.89:0/3081032560 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e80f700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d4815bdd0 0x7f0d481537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4d80d700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d480d1c70 0x7f0d480c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e00e700 1 -- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d4815bdd0 msgr2=0x7f0d481537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e00e700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d4815bdd0 0x7f0d481537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e00e700 1 -- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d480d1c70 msgr2=0x7f0d480c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e00e700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d480d1c70 0x7f0d480c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e00e700 1 -- 172.21.15.89:0/3081032560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d38009550 con 0x7f0d480b3370 2023-11-24T06:21:19.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.448+0000 7f0d4e80f700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d4815bdd0 0x7f0d481537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:19.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.449+0000 7f0d4e00e700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d480cb670 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400b470 tx=0x7f0d4400b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.449+0000 7f0d3effd700 1 -- 172.21.15.89:0/3081032560 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44011830 con 0x7f0d480b3370 2023-11-24T06:21:19.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.449+0000 7f0d3effd700 1 -- 172.21.15.89:0/3081032560 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44011990 con 0x7f0d480b3370 2023-11-24T06:21:19.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.449+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d480ca8b0 con 0x7f0d480b3370 2023-11-24T06:21:19.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.449+0000 7f0d3effd700 1 -- 172.21.15.89:0/3081032560 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44011d40 con 0x7f0d480b3370 2023-11-24T06:21:19.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.449+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d480cae70 con 0x7f0d480b3370 2023-11-24T06:21:19.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.450+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d48067f60 con 0x7f0d480b3370 2023-11-24T06:21:19.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.450+0000 7f0d3effd700 1 -- 172.21.15.89:0/3081032560 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f0d4401a400 con 0x7f0d480b3370 2023-11-24T06:21:19.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.450+0000 7f0d3effd700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d2c06aff0 0x7f0d2c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.451+0000 7f0d4d80d700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d2c06aff0 0x7f0d2c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.451+0000 7f0d3effd700 1 -- 172.21.15.89:0/3081032560 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d440897b0 con 0x7f0d480b3370 2023-11-24T06:21:19.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.451+0000 7f0d4d80d700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d2c06aff0 0x7f0d2c06d4b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009130 tx=0x7f0d38008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.455+0000 7f0d3effd700 1 -- 172.21.15.89:0/3081032560 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d44054750 con 0x7f0d480b3370 2023-11-24T06:21:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.686+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d480c5290 con 0x7f0d480b3370 2023-11-24T06:21:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.686+0000 7f0d3effd700 1 -- 172.21.15.89:0/3081032560 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f0d44015090 con 0x7f0d480b3370 2023-11-24T06:21:19.689 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:19.689 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/351443905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":351443905},{"type":"v1","addr":"172.21.15.123:6834","nonce":351443905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6021,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2890850053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2890850053},{"type":"v1","addr":"172.21.15.89:6836","nonce":2890850053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3912248946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3912248946},{"type":"v1","addr":"172.21.15.89:6837","nonce":3912248946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:20:56.834302+0000","modified":"2023-11-24T06:20:59.846940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6039":{"gid":6039,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2963752713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2963752713},{"type":"v1","addr":"172.21.15.123:6835","nonce":2963752713}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d2c06aff0 msgr2=0x7f0d2c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d2c06aff0 0x7f0d2c06d4b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009130 tx=0x7f0d38008030 comp rx=0 tx=0).stop 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 msgr2=0x7f0d480cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d480cb670 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400b470 tx=0x7f0d4400b830 comp rx=0 tx=0).stop 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 shutdown_connections 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d4815bdd0 0x7f0d481537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d480b3370 0x7f0d480cb670 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f0d2c06aff0 0x7f0d2c06d4b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 --2- 172.21.15.89:0/3081032560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d480d1c70 0x7f0d480c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.688+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 >> 172.21.15.89:0/3081032560 conn(0x7f0d480731a0 msgr2=0x7f0d4815a320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:19.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.689+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 shutdown_connections 2023-11-24T06:21:19.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.689+0000 7f0d50272700 1 -- 172.21.15.89:0/3081032560 wait complete. 2023-11-24T06:21:19.693 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 42 2023-11-24T06:21:19.703 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:21:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.821+0000 7f37915a7700 1 Processor -- start 2023-11-24T06:21:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.821+0000 7f37915a7700 1 -- start start 2023-11-24T06:21:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.822+0000 7f37915a7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c06c710 0x7f378c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.822+0000 7f37915a7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f378c0af4d0 0x7f378c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.822+0000 7f37915a7700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c0ae430 con 0x7f378c15e080 2023-11-24T06:21:19.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.822+0000 7f37915a7700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f378c0af0b0 con 0x7f378c0af4d0 2023-11-24T06:21:19.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.822+0000 7f37915a7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c0b05a0 con 0x7f378c06c710 2023-11-24T06:21:19.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.822+0000 7f378b7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f378c0af4d0 0x7f378c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.822+0000 7f378affd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f378c15e080 0x7f378c06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43046/0 (socket says 172.21.15.89:43046) 2023-11-24T06:21:19.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.822+0000 7f378affd700 1 -- 172.21.15.89:0/4234310273 learned_addr learned my addr 172.21.15.89:0/4234310273 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f378a7fc700 1 --2- 172.21.15.89:0/4234310273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c06c710 0x7f378c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f378b7fe700 1 -- 172.21.15.89:0/4234310273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c06c710 msgr2=0x7f378c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f378b7fe700 1 --2- 172.21.15.89:0/4234310273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c06c710 0x7f378c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f378b7fe700 1 -- 172.21.15.89:0/4234310273 >> v1:172.21.15.89:6789/0 conn(0x7f378c15e080 legacy=0x7f378c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f378b7fe700 1 -- 172.21.15.89:0/4234310273 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c0ae710 con 0x7f378c0af4d0 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f3789ffb700 1 -- 172.21.15.89:0/4234310273 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059129326 0 0) 0x7f378c0ae430 con 0x7f378c15e080 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f378b7fe700 1 --2- 172.21.15.89:0/4234310273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c0af4d0 0x7f378c0afb90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f3784009dc0 tx=0x7f378400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6a93923455c26c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f3789ffb700 1 -- 172.21.15.89:0/4234310273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378400e070 con 0x7f378c0af4d0 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f3789ffb700 1 -- 172.21.15.89:0/4234310273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378400bb90 con 0x7f378c0af4d0 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f3789ffb700 1 -- 172.21.15.89:0/4234310273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784012430 con 0x7f378c0af4d0 2023-11-24T06:21:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f37915a7700 1 -- 172.21.15.89:0/4234310273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c0af4d0 msgr2=0x7f378c0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f37915a7700 1 --2- 172.21.15.89:0/4234310273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c0af4d0 0x7f378c0afb90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f3784009dc0 tx=0x7f378400b330 comp rx=0 tx=0).stop 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f37915a7700 1 -- 172.21.15.89:0/4234310273 shutdown_connections 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f37915a7700 1 --2- 172.21.15.89:0/4234310273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c0af4d0 0x7f378c0afb90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f37915a7700 1 --2- 172.21.15.89:0/4234310273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c06c710 0x7f378c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.823+0000 7f37915a7700 1 -- 172.21.15.89:0/4234310273 >> 172.21.15.89:0/4234310273 conn(0x7f378c071b90 msgr2=0x7f378c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.824+0000 7f37915a7700 1 -- 172.21.15.89:0/4234310273 shutdown_connections 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.824+0000 7f37915a7700 1 -- 172.21.15.89:0/4234310273 wait complete. 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.825+0000 7f37915a7700 1 Processor -- start 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.825+0000 7f37915a7700 1 -- start start 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37915a7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c06c710 0x7f378c0cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37915a7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c0af4d0 0x7f378c1604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37915a7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378c15e080 0x7f378c160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37915a7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c15f4b0 con 0x7f378c06c710 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37915a7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c15f700 con 0x7f378c15e080 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37915a7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c15f9e0 con 0x7f378c0af4d0 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c06c710 0x7f378c0cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c06c710 0x7f378c0cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39548/0 (socket says 172.21.15.89:39548) 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 -- 172.21.15.89:0/2322466661 learned_addr learned my addr 172.21.15.89:0/2322466661 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 -- 172.21.15.89:0/2322466661 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c0af4d0 msgr2=0x7f378c1604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c0af4d0 0x7f378c1604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378b7fe700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378c15e080 0x7f378c160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 -- 172.21.15.89:0/2322466661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378c15e080 msgr2=0x7f378c160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378c15e080 0x7f378c160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 -- 172.21.15.89:0/2322466661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3784009a70 con 0x7f378c06c710 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378b7fe700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378c15e080 0x7f378c160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f378affd700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c06c710 0x7f378c0cad60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f378000c550 tx=0x7f378000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37737fe700 1 -- 172.21.15.89:0/2322466661 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780004c80 con 0x7f378c06c710 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37737fe700 1 -- 172.21.15.89:0/2322466661 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780004e00 con 0x7f378c06c710 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378c15fcc0 con 0x7f378c06c710 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37737fe700 1 -- 172.21.15.89:0/2322466661 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780007310 con 0x7f378c06c710 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.826+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c14bec0 con 0x7f378c06c710 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.827+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378c0753f0 con 0x7f378c06c710 2023-11-24T06:21:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.828+0000 7f37737fe700 1 -- 172.21.15.89:0/2322466661 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3780007470 con 0x7f378c06c710 2023-11-24T06:21:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.828+0000 7f37737fe700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f377806aff0 0x7f377806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.828+0000 7f37737fe700 1 -- 172.21.15.89:0/2322466661 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37800893c0 con 0x7f378c06c710 2023-11-24T06:21:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.828+0000 7f378a7fc700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f377806aff0 0x7f377806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.829+0000 7f378a7fc700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f377806aff0 0x7f377806d4b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f378c06f590 tx=0x7f377c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:19.833+0000 7f37737fe700 1 -- 172.21.15.89:0/2322466661 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37800543f0 con 0x7f378c06c710 2023-11-24T06:21:20.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.024+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f378c067f60 con 0x7f378c06c710 2023-11-24T06:21:20.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.025+0000 7f37737fe700 1 -- 172.21.15.89:0/2322466661 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f3780057a10 con 0x7f378c06c710 2023-11-24T06:21:20.028 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:20.028 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":64,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:20:58.434050+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:20:55.237439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-24T06:20:55.790741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:21:20.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f377806aff0 msgr2=0x7f377806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f377806aff0 0x7f377806d4b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f378c06f590 tx=0x7f377c008040 comp rx=0 tx=0).stop 2023-11-24T06:21:20.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c06c710 msgr2=0x7f378c0cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c06c710 0x7f378c0cad60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f378000c550 tx=0x7f378000c910 comp rx=0 tx=0).stop 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 shutdown_connections 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f377806aff0 0x7f377806d4b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378c15e080 0x7f378c160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f378c06c710 0x7f378c0cad60 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 --2- 172.21.15.89:0/2322466661 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f378c0af4d0 0x7f378c1604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 >> 172.21.15.89:0/2322466661 conn(0x7f378c071b90 msgr2=0x7f378c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 shutdown_connections 2023-11-24T06:21:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.027+0000 7f37915a7700 1 -- 172.21.15.89:0/2322466661 wait complete. 2023-11-24T06:21:20.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T06:21:20.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.161+0000 7fee2c6c3700 1 Processor -- start 2023-11-24T06:21:20.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.161+0000 7fee2c6c3700 1 -- start start 2023-11-24T06:21:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2c6c3700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fee2415bdd0 0x7fee241604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2c6c3700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee240b22d0 con 0x7fee240d1c70 2023-11-24T06:21:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2c6c3700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee240b25b0 con 0x7fee240b3370 2023-11-24T06:21:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2c6c3700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee240b2f50 con 0x7fee2415bdd0 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2a45f700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fee2415bdd0 0x7fee241604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2ac60700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fee240d1c70 0x7fee24165ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43064/0 (socket says 172.21.15.89:43064) 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2a45f700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fee2415bdd0 0x7fee241604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39556/0 (socket says 172.21.15.89:39556) 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2ac60700 1 -- 172.21.15.89:0/3170079281 learned_addr learned my addr 172.21.15.89:0/3170079281 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2a45f700 1 -- 172.21.15.89:0/3170079281 >> v1:172.21.15.123:6789/0 conn(0x7fee240b3370 legacy=0x7fee240b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2a45f700 1 -- 172.21.15.89:0/3170079281 >> v1:172.21.15.89:6789/0 conn(0x7fee240d1c70 legacy=0x7fee24165ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2945d700 1 -- 172.21.15.89:0/3170079281 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784798999 0 0) 0x7fee240b22d0 con 0x7fee240d1c70 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.162+0000 7fee2a45f700 1 -- 172.21.15.89:0/3170079281 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee240b2890 con 0x7fee2415bdd0 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2a45f700 1 --2- 172.21.15.89:0/3170079281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee241604e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fee20009840 tx=0x7fee20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb62c36d949149bc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2945d700 1 -- 172.21.15.89:0/3170079281 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee20004230 con 0x7fee2415bdd0 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2945d700 1 -- 172.21.15.89:0/3170079281 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee20004390 con 0x7fee2415bdd0 2023-11-24T06:21:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2945d700 1 -- 172.21.15.89:0/3170079281 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200036a0 con 0x7fee2415bdd0 2023-11-24T06:21:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2c6c3700 1 -- 172.21.15.89:0/3170079281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 msgr2=0x7fee241604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2c6c3700 1 --2- 172.21.15.89:0/3170079281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee241604e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fee20009840 tx=0x7fee20009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2c6c3700 1 -- 172.21.15.89:0/3170079281 shutdown_connections 2023-11-24T06:21:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2c6c3700 1 --2- 172.21.15.89:0/3170079281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee241604e0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2c6c3700 1 -- 172.21.15.89:0/3170079281 >> 172.21.15.89:0/3170079281 conn(0x7fee240731a0 msgr2=0x7fee2409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2c6c3700 1 -- 172.21.15.89:0/3170079281 shutdown_connections 2023-11-24T06:21:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.163+0000 7fee2c6c3700 1 -- 172.21.15.89:0/3170079281 wait complete. 2023-11-24T06:21:20.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.164+0000 7fee2c6c3700 1 Processor -- start 2023-11-24T06:21:20.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2c6c3700 1 -- start start 2023-11-24T06:21:20.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2c6c3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee240b3370 0x7fee241eef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2c6c3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee240d1c70 0x7fee240cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2c6c3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee240cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2c6c3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee241ef440 con 0x7fee2415bdd0 2023-11-24T06:21:20.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2c6c3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee241f22a0 con 0x7fee240d1c70 2023-11-24T06:21:20.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2c6c3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee241f1990 con 0x7fee240b3370 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2ac60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee240cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2ac60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee240cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39566/0 (socket says 172.21.15.89:39566) 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.165+0000 7fee2ac60700 1 -- 172.21.15.89:0/981795981 learned_addr learned my addr 172.21.15.89:0/981795981 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee29c5e700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee240d1c70 0x7fee240cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2ac60700 1 -- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee240b3370 msgr2=0x7fee241eef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2a45f700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee240b3370 0x7fee241eef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2ac60700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee240b3370 0x7fee241eef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2ac60700 1 -- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee240d1c70 msgr2=0x7fee240cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2ac60700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee240d1c70 0x7fee240cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2ac60700 1 -- 172.21.15.89:0/981795981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200094a0 con 0x7fee2415bdd0 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee29c5e700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee240d1c70 0x7fee240cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:20.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2ac60700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee240cd870 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00ccd0 tx=0x7fee1c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee1affd700 1 -- 172.21.15.89:0/981795981 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c00ada0 con 0x7fee2415bdd0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee1affd700 1 -- 172.21.15.89:0/981795981 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c0070d0 con 0x7fee2415bdd0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee240c9b50 con 0x7fee2415bdd0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee1affd700 1 -- 172.21.15.89:0/981795981 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c007480 con 0x7fee2415bdd0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.166+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee241f1f30 con 0x7fee2415bdd0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.167+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee241f2430 con 0x7fee2415bdd0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.168+0000 7fee1affd700 1 -- 172.21.15.89:0/981795981 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fee1c0075e0 con 0x7fee2415bdd0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.168+0000 7fee1affd700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fee0806aff0 0x7fee0806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.168+0000 7fee1affd700 1 -- 172.21.15.89:0/981795981 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee1c089620 con 0x7fee2415bdd0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.168+0000 7fee2a45f700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fee0806aff0 0x7fee0806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.168+0000 7fee2a45f700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fee0806aff0 0x7fee0806d4b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fee240d1450 tx=0x7fee20013040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:20.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.172+0000 7fee1affd700 1 -- 172.21.15.89:0/981795981 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee1c0545c0 con 0x7fee2415bdd0 2023-11-24T06:21:20.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.409+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fee24067f60 con 0x7fee2415bdd0 2023-11-24T06:21:20.412 INFO:tasks.ceph.mon.a.smithi089.stderr:2023-11-24T06:21:20.409+0000 7f7eccc4c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T06:21:20.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.414+0000 7fee1affd700 1 -- 172.21.15.89:0/981795981 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7fee1c057be0 con 0x7fee2415bdd0 2023-11-24T06:21:20.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fee0806aff0 msgr2=0x7fee0806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fee0806aff0 0x7fee0806d4b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fee240d1450 tx=0x7fee20013040 comp rx=0 tx=0).stop 2023-11-24T06:21:20.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 msgr2=0x7fee240cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee240cd870 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00ccd0 tx=0x7fee1c00a340 comp rx=0 tx=0).stop 2023-11-24T06:21:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 shutdown_connections 2023-11-24T06:21:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee2415bdd0 0x7fee240cd870 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee240b3370 0x7fee241eef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fee0806aff0 0x7fee0806d4b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 --2- 172.21.15.89:0/981795981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee240d1c70 0x7fee240cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 >> 172.21.15.89:0/981795981 conn(0x7fee240731a0 msgr2=0x7fee240aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 shutdown_connections 2023-11-24T06:21:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.416+0000 7fee2c6c3700 1 -- 172.21.15.89:0/981795981 wait complete. 2023-11-24T06:21:20.421 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T06:21:20.423 INFO:tasks.ceph.mds.b.smithi123.stderr: -7255> 2023-11-24T06:20:59.417+0000 7f7efd0b2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:21:20.431 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T06:21:20.457 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2023-11-24T06:21:20.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.552+0000 7f260d9fc700 1 Processor -- start 2023-11-24T06:21:20.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.552+0000 7f260d9fc700 1 -- start start 2023-11-24T06:21:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f260d9fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2608071610 0x7f2608071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f260d9fc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f26080b6fb0 0x7f26080b7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f260d9fc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26080d2a60 con 0x7f260815e090 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f260d9fc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2608072000 con 0x7f2608071610 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f260d9fc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26080d2d40 con 0x7f26080b6fb0 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26067fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2608071610 0x7f2608071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26067fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2608071610 0x7f2608071ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39572/0 (socket says 172.21.15.89:39572) 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26067fc700 1 -- 172.21.15.89:0/3343799466 learned_addr learned my addr 172.21.15.89:0/3343799466 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26077fe700 1 --2- 172.21.15.89:0/3343799466 >> v2:172.21.15.123:3300/0 conn(0x7f26080b6fb0 0x7f26080b7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26067fc700 1 -- 172.21.15.89:0/3343799466 >> v2:172.21.15.123:3300/0 conn(0x7f26080b6fb0 msgr2=0x7f26080b7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26067fc700 1 --2- 172.21.15.89:0/3343799466 >> v2:172.21.15.123:3300/0 conn(0x7f26080b6fb0 0x7f26080b7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26067fc700 1 -- 172.21.15.89:0/3343799466 >> v1:172.21.15.123:6789/0 conn(0x7f260815e090 legacy=0x7f26080d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26067fc700 1 -- 172.21.15.89:0/3343799466 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26080d3020 con 0x7f2608071610 2023-11-24T06:21:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.553+0000 7f26067fc700 1 --2- 172.21.15.89:0/3343799466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2608071610 0x7f2608071ac0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f26080d0020 tx=0x7f25f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b40b90fdfaebd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f2605ffb700 1 -- 172.21.15.89:0/3343799466 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8004230 con 0x7f2608071610 2023-11-24T06:21:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f2605ffb700 1 -- 172.21.15.89:0/3343799466 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f8004390 con 0x7f2608071610 2023-11-24T06:21:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f2605ffb700 1 -- 172.21.15.89:0/3343799466 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f80036a0 con 0x7f2608071610 2023-11-24T06:21:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f260d9fc700 1 -- 172.21.15.89:0/3343799466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2608071610 msgr2=0x7f2608071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3343799466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2608071610 0x7f2608071ac0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f26080d0020 tx=0x7f25f8009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f260d9fc700 1 -- 172.21.15.89:0/3343799466 shutdown_connections 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3343799466 >> v2:172.21.15.123:3300/0 conn(0x7f26080b6fb0 0x7f26080b7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3343799466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2608071610 0x7f2608071ac0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f260d9fc700 1 -- 172.21.15.89:0/3343799466 >> 172.21.15.89:0/3343799466 conn(0x7f260806b410 msgr2=0x7f260806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f260d9fc700 1 -- 172.21.15.89:0/3343799466 shutdown_connections 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.554+0000 7f260d9fc700 1 -- 172.21.15.89:0/3343799466 wait complete. 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.555+0000 7f260d9fc700 1 Processor -- start 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f260d9fc700 1 -- start start 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f260d9fc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2608071610 0x7f260815bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f260d9fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26080b6fb0 0x7f260814dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f260d9fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f260815e090 0x7f26080b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f260d9fc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260815c1c0 con 0x7f26080b6fb0 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f260d9fc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2608150200 con 0x7f260815e090 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f260d9fc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260814e680 con 0x7f2608071610 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f26067fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26080b6fb0 0x7f260814dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f26067fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26080b6fb0 0x7f260814dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39578/0 (socket says 172.21.15.89:39578) 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.556+0000 7f26067fc700 1 -- 172.21.15.89:0/3855261843 learned_addr learned my addr 172.21.15.89:0/3855261843 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f26077fe700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f260815e090 0x7f26080b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f2606ffd700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2608071610 0x7f260815bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f26067fc700 1 -- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2608071610 msgr2=0x7f260815bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f26067fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2608071610 0x7f260815bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f26067fc700 1 -- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f260815e090 msgr2=0x7f26080b75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f26067fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f260815e090 0x7f26080b75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f26067fc700 1 -- 172.21.15.89:0/3855261843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8009550 con 0x7f26080b6fb0 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f26077fe700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f260815e090 0x7f26080b75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f26067fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26080b6fb0 0x7f260814dc00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f26080723c0 tx=0x7f25f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f25effff700 1 -- 172.21.15.89:0/3855261843 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8003d40 con 0x7f26080b6fb0 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f25effff700 1 -- 172.21.15.89:0/3855261843 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f801f070 con 0x7f26080b6fb0 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260814e960 con 0x7f26080b6fb0 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f25effff700 1 -- 172.21.15.89:0/3855261843 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8011870 con 0x7f26080b6fb0 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.557+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2608150730 con 0x7f26080b6fb0 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.558+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26081508c0 con 0x7f26080b6fb0 2023-11-24T06:21:20.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.558+0000 7f25effff700 1 -- 172.21.15.89:0/3855261843 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f25f8021030 con 0x7f26080b6fb0 2023-11-24T06:21:20.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.559+0000 7f25effff700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f25f406aff0 0x7f25f406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:20.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.559+0000 7f25effff700 1 -- 172.21.15.89:0/3855261843 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f25f808ad30 con 0x7f26080b6fb0 2023-11-24T06:21:20.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.559+0000 7f2606ffd700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f25f406aff0 0x7f25f406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:20.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.559+0000 7f2606ffd700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f25f406aff0 0x7f25f406d4b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009870 tx=0x7f25fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:20.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.563+0000 7f25effff700 1 -- 172.21.15.89:0/3855261843 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25f8055c70 con 0x7f26080b6fb0 2023-11-24T06:21:20.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:20.800+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2608068fb0 con 0x7f26080b6fb0 2023-11-24T06:21:21.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.418+0000 7f25effff700 1 -- 172.21.15.89:0/3855261843 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f25f8059290 con 0x7f26080b6fb0 2023-11-24T06:21:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f25f406aff0 msgr2=0x7f25f406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f25f406aff0 0x7f25f406d4b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009870 tx=0x7f25fc008040 comp rx=0 tx=0).stop 2023-11-24T06:21:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26080b6fb0 msgr2=0x7f260814dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26080b6fb0 0x7f260814dc00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f26080723c0 tx=0x7f25f8004750 comp rx=0 tx=0).stop 2023-11-24T06:21:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 shutdown_connections 2023-11-24T06:21:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f25f406aff0 0x7f25f406d4b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f260815e090 0x7f26080b75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2608071610 0x7f260815bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 --2- 172.21.15.89:0/3855261843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26080b6fb0 0x7f260814dc00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 >> 172.21.15.89:0/3855261843 conn(0x7f260806b410 msgr2=0x7f26080d3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:21.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 shutdown_connections 2023-11-24T06:21:21.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.421+0000 7f260d9fc700 1 -- 172.21.15.89:0/3855261843 wait complete. 2023-11-24T06:21:21.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T06:21:21.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.562+0000 7f8656111700 1 Processor -- start 2023-11-24T06:21:21.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.562+0000 7f8656111700 1 -- start start 2023-11-24T06:21:21.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f8656111700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:21.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f8656111700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f86500d1b00 0x7f86500d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:21.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f8656111700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86500d2490 con 0x7f865015e260 2023-11-24T06:21:21.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f8656111700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865006b580 con 0x7f86500d1b00 2023-11-24T06:21:21.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f8656111700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86500d02d0 con 0x7f865015de80 2023-11-24T06:21:21.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f864ffff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f86500d1b00 0x7f86500d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f864f7fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f864effd700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f865015e260 0x7f865006ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:46910/0 (socket says 172.21.15.89:46910) 2023-11-24T06:21:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f864f7fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:48460/0 (socket says 172.21.15.89:48460) 2023-11-24T06:21:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f864effd700 1 -- 172.21.15.89:0/94282422 learned_addr learned my addr 172.21.15.89:0/94282422 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.563+0000 7f864ffff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f86500d1b00 0x7f86500d1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:54700/0 (socket says 172.21.15.89:54700) 2023-11-24T06:21:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864ffff700 1 -- 172.21.15.89:0/94282422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 msgr2=0x7f86500cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864ffff700 1 --2- 172.21.15.89:0/94282422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864ffff700 1 -- 172.21.15.89:0/94282422 >> v1:172.21.15.123:6789/0 conn(0x7f865015e260 legacy=0x7f865006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864ffff700 1 -- 172.21.15.89:0/94282422 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865006f810 con 0x7f86500d1b00 2023-11-24T06:21:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864e7fc700 1 -- 172.21.15.89:0/94282422 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294230634 0 0) 0x7f8640001110 con 0x7f865015e260 2023-11-24T06:21:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864f7fe700 1 --2- 172.21.15.89:0/94282422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:21:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864ffff700 1 --2- 172.21.15.89:0/94282422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86500d1b00 0x7f86500d1f50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8648009d90 tx=0x7f864800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64e3ce6e3c7da587 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864e7fc700 1 -- 172.21.15.89:0/94282422 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864800e070 con 0x7f86500d1b00 2023-11-24T06:21:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864e7fc700 1 -- 172.21.15.89:0/94282422 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864800bb20 con 0x7f86500d1b00 2023-11-24T06:21:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.564+0000 7f864e7fc700 1 -- 172.21.15.89:0/94282422 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8648012430 con 0x7f86500d1b00 2023-11-24T06:21:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.565+0000 7f8656111700 1 -- 172.21.15.89:0/94282422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86500d1b00 msgr2=0x7f86500d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.565+0000 7f8656111700 1 --2- 172.21.15.89:0/94282422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86500d1b00 0x7f86500d1f50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8648009d90 tx=0x7f864800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.565+0000 7f8656111700 1 -- 172.21.15.89:0/94282422 shutdown_connections 2023-11-24T06:21:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.565+0000 7f8656111700 1 --2- 172.21.15.89:0/94282422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86500d1b00 0x7f86500d1f50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.565+0000 7f8656111700 1 --2- 172.21.15.89:0/94282422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.565+0000 7f8656111700 1 -- 172.21.15.89:0/94282422 >> 172.21.15.89:0/94282422 conn(0x7f865009aa00 msgr2=0x7f865009ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:21.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.565+0000 7f8656111700 1 -- 172.21.15.89:0/94282422 shutdown_connections 2023-11-24T06:21:21.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.565+0000 7f8656111700 1 -- 172.21.15.89:0/94282422 wait complete. 2023-11-24T06:21:21.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.566+0000 7f8656111700 1 Processor -- start 2023-11-24T06:21:21.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.566+0000 7f8656111700 1 -- start start 2023-11-24T06:21:21.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f8656111700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86500d1b00 0x7f86501511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:21.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f8656111700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:21.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f8656111700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f865015e260 0x7f8650159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f8656111700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650151720 con 0x7f86500d1b00 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f8656111700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501585a0 con 0x7f865015e260 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f8656111700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86500c6ef0 con 0x7f865015de80 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86500d1b00 0x7f86501511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86500d1b00 0x7f86501511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39590/0 (socket says 172.21.15.89:39590) 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 -- 172.21.15.89:0/2259561239 learned_addr learned my addr 172.21.15.89:0/2259561239 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864effd700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864ffff700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f865015e260 0x7f8650159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 -- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 msgr2=0x7f86500c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 -- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f865015e260 msgr2=0x7f8650159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f865015e260 0x7f8650159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 -- 172.21.15.89:0/2259561239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86380094a0 con 0x7f86500d1b00 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864ffff700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f865015e260 0x7f8650159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864effd700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f864f7fe700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86500d1b00 0x7f86501511e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f863800ebe0 tx=0x7f8638009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.567+0000 7f8647fff700 1 -- 172.21.15.89:0/2259561239 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863800c910 con 0x7f86500d1b00 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.568+0000 7f8647fff700 1 -- 172.21.15.89:0/2259561239 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863800ca70 con 0x7f86500d1b00 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.568+0000 7f8647fff700 1 -- 172.21.15.89:0/2259561239 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863800cde0 con 0x7f86500d1b00 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.568+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8648009a70 con 0x7f86500d1b00 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.568+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86500c74b0 con 0x7f86500d1b00 2023-11-24T06:21:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.568+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86500cc0a0 con 0x7f86500d1b00 2023-11-24T06:21:21.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.569+0000 7f8647fff700 1 -- 172.21.15.89:0/2259561239 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f8638013470 con 0x7f86500d1b00 2023-11-24T06:21:21.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.569+0000 7f8647fff700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f863006aff0 0x7f863006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:21.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.569+0000 7f8647fff700 1 -- 172.21.15.89:0/2259561239 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f86380898d0 con 0x7f86500d1b00 2023-11-24T06:21:21.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.569+0000 7f864effd700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f863006aff0 0x7f863006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:21.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.570+0000 7f864effd700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f863006aff0 0x7f863006d4b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8640000f60 tx=0x7f8640007ae0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:21.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.574+0000 7f8647fff700 1 -- 172.21.15.89:0/2259561239 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86380548a0 con 0x7f86500d1b00 2023-11-24T06:21:21.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:21.765+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86500d1920 con 0x7f86500d1b00 2023-11-24T06:21:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.431+0000 7f8647fff700 1 -- 172.21.15.89:0/2259561239 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v66) v1 ==== 181+0+0 (secure 0 0 0) 0x7f86380170d0 con 0x7f86500d1b00 2023-11-24T06:21:22.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f863006aff0 msgr2=0x7f863006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:22.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f863006aff0 0x7f863006d4b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8640000f60 tx=0x7f8640007ae0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86500d1b00 msgr2=0x7f86501511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:22.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86500d1b00 0x7f86501511e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f863800ebe0 tx=0x7f8638009ec0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 shutdown_connections 2023-11-24T06:21:22.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f863006aff0 0x7f863006d4b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f865015e260 0x7f8650159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86500d1b00 0x7f86501511e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 --2- 172.21.15.89:0/2259561239 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f865015de80 0x7f86500c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 >> 172.21.15.89:0/2259561239 conn(0x7f865009aa00 msgr2=0x7f865009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:22.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 shutdown_connections 2023-11-24T06:21:22.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.433+0000 7f8656111700 1 -- 172.21.15.89:0/2259561239 wait complete. 2023-11-24T06:21:22.437 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2023-11-24T06:21:22.452 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T06:21:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.574+0000 7fed58b79700 1 Processor -- start 2023-11-24T06:21:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.574+0000 7fed58b79700 1 -- start start 2023-11-24T06:21:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.575+0000 7fed58b79700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed5406e830 0x7fed5406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.575+0000 7fed58b79700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5406f1c0 con 0x7fed54144600 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.575+0000 7fed58b79700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5406c9b0 con 0x7fed5406f480 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.575+0000 7fed58b79700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5406aff0 con 0x7fed5406e830 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.575+0000 7fed52d9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed5406e830 0x7fed5406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.575+0000 7fed5259c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fed54144600 0x7fed540b4ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43102/0 (socket says 172.21.15.89:43102) 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.575+0000 7fed5259c700 1 -- 172.21.15.89:0/1048265100 learned_addr learned my addr 172.21.15.89:0/1048265100 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.576+0000 7fed52d9d700 1 -- 172.21.15.89:0/1048265100 >> v1:172.21.15.123:6789/0 conn(0x7fed5406f480 legacy=0x7fed5406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.576+0000 7fed52d9d700 1 -- 172.21.15.89:0/1048265100 >> v1:172.21.15.89:6789/0 conn(0x7fed54144600 legacy=0x7fed540b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.576+0000 7fed52d9d700 1 -- 172.21.15.89:0/1048265100 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5406b200 con 0x7fed5406e830 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.576+0000 7fed5159a700 1 -- 172.21.15.89:0/1048265100 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955433529 0 0) 0x7fed48001110 con 0x7fed54144600 2023-11-24T06:21:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.576+0000 7fed52d9d700 1 --2- 172.21.15.89:0/1048265100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5406e830 0x7fed5406ec80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009d90 tx=0x7fed4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f2998af4b5fb6f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.576+0000 7fed5159a700 1 -- 172.21.15.89:0/1048265100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c00e070 con 0x7fed5406e830 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.576+0000 7fed5159a700 1 -- 172.21.15.89:0/1048265100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c00bbd0 con 0x7fed5406e830 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.576+0000 7fed5159a700 1 -- 172.21.15.89:0/1048265100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c012430 con 0x7fed5406e830 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.577+0000 7fed58b79700 1 -- 172.21.15.89:0/1048265100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5406e830 msgr2=0x7fed5406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.577+0000 7fed58b79700 1 --2- 172.21.15.89:0/1048265100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5406e830 0x7fed5406ec80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009d90 tx=0x7fed4c00b330 comp rx=0 tx=0).stop 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.577+0000 7fed58b79700 1 -- 172.21.15.89:0/1048265100 shutdown_connections 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.577+0000 7fed58b79700 1 --2- 172.21.15.89:0/1048265100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5406e830 0x7fed5406ec80 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.577+0000 7fed58b79700 1 -- 172.21.15.89:0/1048265100 >> 172.21.15.89:0/1048265100 conn(0x7fed540d0b40 msgr2=0x7fed540d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.577+0000 7fed58b79700 1 -- 172.21.15.89:0/1048265100 shutdown_connections 2023-11-24T06:21:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.577+0000 7fed58b79700 1 -- 172.21.15.89:0/1048265100 wait complete. 2023-11-24T06:21:22.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.578+0000 7fed58b79700 1 Processor -- start 2023-11-24T06:21:22.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.578+0000 7fed58b79700 1 -- start start 2023-11-24T06:21:22.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed58b79700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed5406e830 0x7fed541f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed58b79700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed5406f480 0x7fed541f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed58b79700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed54144600 0x7fed541f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed58b79700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54133280 con 0x7fed54144600 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed58b79700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541334d0 con 0x7fed5406f480 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed58b79700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed541337b0 con 0x7fed5406e830 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed54144600 0x7fed541f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed54144600 0x7fed541f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39608/0 (socket says 172.21.15.89:39608) 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 -- 172.21.15.89:0/2308370841 learned_addr learned my addr 172.21.15.89:0/2308370841 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 -- 172.21.15.89:0/2308370841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed5406e830 msgr2=0x7fed541f4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed5406e830 0x7fed541f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 -- 172.21.15.89:0/2308370841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed5406f480 msgr2=0x7fed541f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed51d9b700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed5406f480 0x7fed541f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed5406f480 0x7fed541f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 -- 172.21.15.89:0/2308370841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c009a70 con 0x7fed54144600 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed52d9d700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed54144600 0x7fed541f3e10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fed4c014040 tx=0x7fed4c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.579+0000 7fed42ffd700 1 -- 172.21.15.89:0/2308370841 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c01b810 con 0x7fed54144600 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.580+0000 7fed42ffd700 1 -- 172.21.15.89:0/2308370841 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c020070 con 0x7fed54144600 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.580+0000 7fed42ffd700 1 -- 172.21.15.89:0/2308370841 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c01bbb0 con 0x7fed54144600 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.580+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed54133a90 con 0x7fed54144600 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.580+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed541f3480 con 0x7fed54144600 2023-11-24T06:21:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.580+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed54067f60 con 0x7fed54144600 2023-11-24T06:21:22.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.581+0000 7fed42ffd700 1 -- 172.21.15.89:0/2308370841 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fed4c01bd10 con 0x7fed54144600 2023-11-24T06:21:22.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.581+0000 7fed42ffd700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed3406aff0 0x7fed3406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:22.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.581+0000 7fed5259c700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed3406aff0 0x7fed3406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:22.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.581+0000 7fed42ffd700 1 -- 172.21.15.89:0/2308370841 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fed4c022070 con 0x7fed54144600 2023-11-24T06:21:22.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.582+0000 7fed5259c700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed3406aff0 0x7fed3406d4b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fed48000c00 tx=0x7fed48007a50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:22.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.586+0000 7fed42ffd700 1 -- 172.21.15.89:0/2308370841 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed4c05a380 con 0x7fed54144600 2023-11-24T06:21:22.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:22.782+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fed5406e0e0 con 0x7fed54144600 2023-11-24T06:21:23.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.432+0000 7fed42ffd700 1 -- 172.21.15.89:0/2308370841 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v67) v1 ==== 169+0+0 (secure 0 0 0) 0x7fed4c02c0d0 con 0x7fed54144600 2023-11-24T06:21:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed3406aff0 msgr2=0x7fed3406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed3406aff0 0x7fed3406d4b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fed48000c00 tx=0x7fed48007a50 comp rx=0 tx=0).stop 2023-11-24T06:21:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed54144600 msgr2=0x7fed541f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed54144600 0x7fed541f3e10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fed4c014040 tx=0x7fed4c004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 shutdown_connections 2023-11-24T06:21:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed54144600 0x7fed541f3e10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed5406e830 0x7fed541f4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fed3406aff0 0x7fed3406d4b0 secure :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fed48000c00 tx=0x7fed48007a50 comp rx=0 tx=0).stop 2023-11-24T06:21:23.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 --2- 172.21.15.89:0/2308370841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed5406f480 0x7fed541f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 >> 172.21.15.89:0/2308370841 conn(0x7fed540d0b40 msgr2=0x7fed540ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:23.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.436+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 shutdown_connections 2023-11-24T06:21:23.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.437+0000 7fed58b79700 1 -- 172.21.15.89:0/2308370841 wait complete. 2023-11-24T06:21:23.440 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2023-11-24T06:21:23.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T06:21:23.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.577+0000 7f2eeff23700 1 Processor -- start 2023-11-24T06:21:23.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.577+0000 7f2eeff23700 1 -- start start 2023-11-24T06:21:23.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eeff23700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2ee815bf50 0x7f2ee80ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eeff23700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee80c6c30 0x7f2ee80cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eeff23700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee80d0140 con 0x7f2ee80ce830 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eeff23700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee815c330 con 0x7f2ee815bf50 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eeff23700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80cf3c0 con 0x7f2ee80c6c30 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eedcbf700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2ee815bf50 0x7f2ee80ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eed4be700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2ee80ce830 0x7f2ee80cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43116/0 (socket says 172.21.15.89:43116) 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eedcbf700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2ee815bf50 0x7f2ee80ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39612/0 (socket says 172.21.15.89:39612) 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eed4be700 1 -- 172.21.15.89:0/380198984 learned_addr learned my addr 172.21.15.89:0/380198984 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eedcbf700 1 -- 172.21.15.89:0/380198984 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee80c6c30 msgr2=0x7f2ee80cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eedcbf700 1 --2- 172.21.15.89:0/380198984 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee80c6c30 0x7f2ee80cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eedcbf700 1 -- 172.21.15.89:0/380198984 >> v1:172.21.15.89:6789/0 conn(0x7f2ee80ce830 legacy=0x7f2ee80cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eeccbd700 1 -- 172.21.15.89:0/380198984 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037420462 0 0) 0x7f2ee80d0140 con 0x7f2ee80ce830 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.578+0000 7f2eedcbf700 1 -- 172.21.15.89:0/380198984 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80d03e0 con 0x7f2ee815bf50 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eedcbf700 1 --2- 172.21.15.89:0/380198984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee815bf50 0x7f2ee80ce2f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009840 tx=0x7f2ed8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4f629db86bd8fd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeccbd700 1 -- 172.21.15.89:0/380198984 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8011830 con 0x7f2ee815bf50 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeccbd700 1 -- 172.21.15.89:0/380198984 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8011990 con 0x7f2ee815bf50 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeccbd700 1 -- 172.21.15.89:0/380198984 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8011d20 con 0x7f2ee815bf50 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeff23700 1 -- 172.21.15.89:0/380198984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee815bf50 msgr2=0x7f2ee80ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeff23700 1 --2- 172.21.15.89:0/380198984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee815bf50 0x7f2ee80ce2f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009840 tx=0x7f2ed8009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeff23700 1 -- 172.21.15.89:0/380198984 shutdown_connections 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeff23700 1 --2- 172.21.15.89:0/380198984 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee80c6c30 0x7f2ee80cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeff23700 1 --2- 172.21.15.89:0/380198984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee815bf50 0x7f2ee80ce2f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeff23700 1 -- 172.21.15.89:0/380198984 >> 172.21.15.89:0/380198984 conn(0x7f2ee80b3780 msgr2=0x7f2ee80b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeff23700 1 -- 172.21.15.89:0/380198984 shutdown_connections 2023-11-24T06:21:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.579+0000 7f2eeff23700 1 -- 172.21.15.89:0/380198984 wait complete. 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.581+0000 7f2eeff23700 1 Processor -- start 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.581+0000 7f2eeff23700 1 -- start start 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eeff23700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee80c6c30 0x7f2ee80b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eeff23700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee80ce830 0x7f2ee80aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eeff23700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee815bf50 0x7f2ee80aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eeff23700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80b71c0 con 0x7f2ee80ce830 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eeff23700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8146ba0 con 0x7f2ee815bf50 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eeff23700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8146e80 con 0x7f2ee80c6c30 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee80ce830 0x7f2ee80aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee80ce830 0x7f2ee80aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39618/0 (socket says 172.21.15.89:39618) 2023-11-24T06:21:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 -- 172.21.15.89:0/4253467382 learned_addr learned my addr 172.21.15.89:0/4253467382 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 -- 172.21.15.89:0/4253467382 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee80c6c30 msgr2=0x7f2ee80b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee80c6c30 0x7f2ee80b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 -- 172.21.15.89:0/4253467382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee815bf50 msgr2=0x7f2ee80aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee815bf50 0x7f2ee80aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 -- 172.21.15.89:0/4253467382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed80094a0 con 0x7f2ee80ce830 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eed4be700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee80ce830 0x7f2ee80aea60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40011e0 tx=0x7f2ee400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2ede7fc700 1 -- 172.21.15.89:0/4253467382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4004c80 con 0x7f2ee80ce830 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2ede7fc700 1 -- 172.21.15.89:0/4253467382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee4004e00 con 0x7f2ee80ce830 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.582+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee8147160 con 0x7f2ee80ce830 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.583+0000 7f2ede7fc700 1 -- 172.21.15.89:0/4253467382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4007310 con 0x7f2ee80ce830 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.583+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee80b29e0 con 0x7f2ee80ce830 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.583+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee806e9b0 con 0x7f2ee80ce830 2023-11-24T06:21:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.584+0000 7f2ede7fc700 1 -- 172.21.15.89:0/4253467382 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f2ee4007470 con 0x7f2ee80ce830 2023-11-24T06:21:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.584+0000 7f2ede7fc700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2ecc06aff0 0x7f2ecc06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.584+0000 7f2ede7fc700 1 -- 172.21.15.89:0/4253467382 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2ee4089200 con 0x7f2ee80ce830 2023-11-24T06:21:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.584+0000 7f2eedcbf700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2ecc06aff0 0x7f2ecc06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.585+0000 7f2eedcbf700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2ecc06aff0 0x7f2ecc06d4b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8000c00 tx=0x7f2ed8010000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:23.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.589+0000 7f2ede7fc700 1 -- 172.21.15.89:0/4253467382 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ee40544a0 con 0x7f2ee80ce830 2023-11-24T06:21:23.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.783+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2ee8068fb0 con 0x7f2ee80ce830 2023-11-24T06:21:23.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.786+0000 7f2ede7fc700 1 -- 172.21.15.89:0/4253467382 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2ee4057ac0 con 0x7f2ee80ce830 2023-11-24T06:21:23.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2ecc06aff0 msgr2=0x7f2ecc06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2ecc06aff0 0x7f2ecc06d4b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8000c00 tx=0x7f2ed8010000 comp rx=0 tx=0).stop 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee80ce830 msgr2=0x7f2ee80aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee80ce830 0x7f2ee80aea60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40011e0 tx=0x7f2ee400cae0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 shutdown_connections 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f2ecc06aff0 0x7f2ecc06d4b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee815bf50 0x7f2ee80aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee80c6c30 0x7f2ee80b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 --2- 172.21.15.89:0/4253467382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee80ce830 0x7f2ee80aea60 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 >> 172.21.15.89:0/4253467382 conn(0x7f2ee80b3780 msgr2=0x7f2ee80c7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 shutdown_connections 2023-11-24T06:21:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.788+0000 7f2eeff23700 1 -- 172.21.15.89:0/4253467382 wait complete. 2023-11-24T06:21:23.803 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_info_pending_subvol_deletions' 2023-11-24T06:21:23.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.921+0000 7f92e4f05700 1 Processor -- start 2023-11-24T06:21:23.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.921+0000 7f92e4f05700 1 -- start start 2023-11-24T06:21:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.922+0000 7f92e4f05700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92e006c710 0x7f92e00cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.922+0000 7f92e4f05700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f92e00b49a0 0x7f92e00d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.922+0000 7f92e4f05700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e00d09a0 con 0x7f92e0152cc0 2023-11-24T06:21:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.922+0000 7f92e4f05700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e015c4b0 con 0x7f92e006c710 2023-11-24T06:21:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.922+0000 7f92e4f05700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e015c790 con 0x7f92e00b49a0 2023-11-24T06:21:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.922+0000 7f92ddd9b700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92e006c710 0x7f92e00cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.922+0000 7f92de59c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f92e0152cc0 0x7f92e006c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43130/0 (socket says 172.21.15.89:43130) 2023-11-24T06:21:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.922+0000 7f92de59c700 1 -- 172.21.15.89:0/1671781483 learned_addr learned my addr 172.21.15.89:0/1671781483 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92ded9d700 1 --2- 172.21.15.89:0/1671781483 >> v2:172.21.15.123:3300/0 conn(0x7f92e00b49a0 0x7f92e00d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92ddd9b700 1 -- 172.21.15.89:0/1671781483 >> v2:172.21.15.123:3300/0 conn(0x7f92e00b49a0 msgr2=0x7f92e00d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92ddd9b700 1 --2- 172.21.15.89:0/1671781483 >> v2:172.21.15.123:3300/0 conn(0x7f92e00b49a0 0x7f92e00d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92ddd9b700 1 -- 172.21.15.89:0/1671781483 >> v1:172.21.15.89:6789/0 conn(0x7f92e0152cc0 legacy=0x7f92e006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92ddd9b700 1 -- 172.21.15.89:0/1671781483 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e015ca70 con 0x7f92e006c710 2023-11-24T06:21:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92dd59a700 1 -- 172.21.15.89:0/1671781483 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212427767 0 0) 0x7f92e00d09a0 con 0x7f92e0152cc0 2023-11-24T06:21:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92ddd9b700 1 --2- 172.21.15.89:0/1671781483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e006c710 0x7f92e00cff20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f92e00b6300 tx=0x7f92d0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dced34a23e83d194 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92dd59a700 1 -- 172.21.15.89:0/1671781483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0011830 con 0x7f92e006c710 2023-11-24T06:21:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92dd59a700 1 -- 172.21.15.89:0/1671781483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d0011990 con 0x7f92e006c710 2023-11-24T06:21:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.923+0000 7f92dd59a700 1 -- 172.21.15.89:0/1671781483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0011d20 con 0x7f92e006c710 2023-11-24T06:21:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.924+0000 7f92e4f05700 1 -- 172.21.15.89:0/1671781483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e006c710 msgr2=0x7f92e00cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.924+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1671781483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e006c710 0x7f92e00cff20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f92e00b6300 tx=0x7f92d0009b20 comp rx=0 tx=0).stop 2023-11-24T06:21:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.924+0000 7f92e4f05700 1 -- 172.21.15.89:0/1671781483 shutdown_connections 2023-11-24T06:21:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.924+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1671781483 >> v2:172.21.15.123:3300/0 conn(0x7f92e00b49a0 0x7f92e00d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.924+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1671781483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e006c710 0x7f92e00cff20 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.924+0000 7f92e4f05700 1 -- 172.21.15.89:0/1671781483 >> 172.21.15.89:0/1671781483 conn(0x7f92e0071b90 msgr2=0x7f92e0071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.924+0000 7f92e4f05700 1 -- 172.21.15.89:0/1671781483 shutdown_connections 2023-11-24T06:21:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.924+0000 7f92e4f05700 1 -- 172.21.15.89:0/1671781483 wait complete. 2023-11-24T06:21:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.925+0000 7f92e4f05700 1 Processor -- start 2023-11-24T06:21:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.925+0000 7f92e4f05700 1 -- start start 2023-11-24T06:21:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92e4f05700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e006c710 0x7f92e0147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92e4f05700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e00b49a0 0x7f92e006e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92e4f05700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e0152cc0 0x7f92e006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92e4f05700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e0147aa0 con 0x7f92e00b49a0 2023-11-24T06:21:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92e4f05700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e006e8c0 con 0x7f92e0152cc0 2023-11-24T06:21:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92e4f05700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e006eba0 con 0x7f92e006c710 2023-11-24T06:21:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e00b49a0 0x7f92e006e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e00b49a0 0x7f92e006e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39632/0 (socket says 172.21.15.89:39632) 2023-11-24T06:21:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 -- 172.21.15.89:0/1745191691 learned_addr learned my addr 172.21.15.89:0/1745191691 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:23.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ded9d700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e0152cc0 0x7f92e006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92de59c700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e006c710 0x7f92e0147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 -- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e006c710 msgr2=0x7f92e0147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e006c710 0x7f92e0147560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 -- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e0152cc0 msgr2=0x7f92e006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:23.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e0152cc0 0x7f92e006aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:23.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 -- 172.21.15.89:0/1745191691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00094a0 con 0x7f92e00b49a0 2023-11-24T06:21:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92de59c700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e006c710 0x7f92e0147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.926+0000 7f92ddd9b700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e00b49a0 0x7f92e006e070 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f92d00090b0 tx=0x7f92d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.927+0000 7f92ceffd700 1 -- 172.21.15.89:0/1745191691 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d00038a0 con 0x7f92e00b49a0 2023-11-24T06:21:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.927+0000 7f92ceffd700 1 -- 172.21.15.89:0/1745191691 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d001f070 con 0x7f92e00b49a0 2023-11-24T06:21:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.927+0000 7f92ceffd700 1 -- 172.21.15.89:0/1745191691 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0003c20 con 0x7f92e00b49a0 2023-11-24T06:21:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.927+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e006ee80 con 0x7f92e00b49a0 2023-11-24T06:21:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.927+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e006f440 con 0x7f92e00b49a0 2023-11-24T06:21:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.928+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e00753f0 con 0x7f92e00b49a0 2023-11-24T06:21:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.928+0000 7f92ceffd700 1 -- 172.21.15.89:0/1745191691 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f92d001a430 con 0x7f92e00b49a0 2023-11-24T06:21:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.928+0000 7f92ceffd700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f92c006aff0 0x7f92c006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.928+0000 7f92ceffd700 1 -- 172.21.15.89:0/1745191691 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f92d0021070 con 0x7f92e00b49a0 2023-11-24T06:21:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.928+0000 7f92de59c700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f92c006aff0 0x7f92c006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.929+0000 7f92de59c700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f92c006aff0 0x7f92c006d4b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f92d4007fd0 tx=0x7f92d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:23.933+0000 7f92ceffd700 1 -- 172.21.15.89:0/1745191691 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92d0056160 con 0x7f92e00b49a0 2023-11-24T06:21:24.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.201+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_info_pending_subvol_deletions"]} v 0) v1 -- 0x7f92e0067f60 con 0x7f92e00b49a0 2023-11-24T06:21:24.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.436+0000 7f92ceffd700 1 -- 172.21.15.89:0/1745191691 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_volume_info_pending_subvol_deletions"]}]=0 v172) v1 ==== 158+0+0 (secure 0 0 0) 0x7f92d002b0e0 con 0x7f92e00b49a0 2023-11-24T06:21:24.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f92c006aff0 msgr2=0x7f92c006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:24.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f92c006aff0 0x7f92c006d4b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f92d4007fd0 tx=0x7f92d4009040 comp rx=0 tx=0).stop 2023-11-24T06:21:24.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e00b49a0 msgr2=0x7f92e006e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:24.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e00b49a0 0x7f92e006e070 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f92d00090b0 tx=0x7f92d0004750 comp rx=0 tx=0).stop 2023-11-24T06:21:24.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 shutdown_connections 2023-11-24T06:21:24.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f92c006aff0 0x7f92c006d4b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:24.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e0152cc0 0x7f92e006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e006c710 0x7f92e0147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 --2- 172.21.15.89:0/1745191691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e00b49a0 0x7f92e006e070 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 >> 172.21.15.89:0/1745191691 conn(0x7f92e0071b90 msgr2=0x7f92e00b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 shutdown_connections 2023-11-24T06:21:24.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:24.439+0000 7f92e4f05700 1 -- 172.21.15.89:0/1745191691 wait complete. 2023-11-24T06:21:24.456 INFO:tasks.cephfs_test_runner:Tests the pending_subvolume_deletions in 'fs volume info' command ... ERROR 2023-11-24T06:21:24.456 INFO:tasks.cephfs_test_runner: 2023-11-24T06:21:24.456 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-24T06:21:24.457 INFO:tasks.cephfs_test_runner:ERROR: test_volume_info_pending_subvol_deletions (tasks.cephfs.test_volumes.TestVolumes) 2023-11-24T06:21:24.457 INFO:tasks.cephfs_test_runner:Tests the pending_subvolume_deletions in 'fs volume info' command 2023-11-24T06:21:24.457 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T06:21:24.457 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-24T06:21:24.457 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs/test_volumes.py", line 568, in test_volume_info_pending_subvol_deletions 2023-11-24T06:21:24.457 INFO:tasks.cephfs_test_runner: self._do_subvolume_io(subvolname, number_of_files=3000, file_size=0) 2023-11-24T06:21:24.457 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs/test_volumes.py", line 278, in _do_subvolume_io 2023-11-24T06:21:24.458 INFO:tasks.cephfs_test_runner: filename = "{0}.{1}".format(TestVolumes.TEST_FILE_NAME_PREFIX, i) 2023-11-24T06:21:24.458 INFO:tasks.cephfs_test_runner:AttributeError: type object 'TestVolumes' has no attribute 'TEST_FILE_NAME_PREFIX' 2023-11-24T06:21:24.458 INFO:tasks.cephfs_test_runner: 2023-11-24T06:21:24.458 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T06:21:24.458 INFO:tasks.cephfs_test_runner:Ran 3 tests in 126.041s 2023-11-24T06:21:24.458 INFO:tasks.cephfs_test_runner: 2023-11-24T06:21:24.458 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2023-11-24T06:21:24.459 INFO:tasks.cephfs_test_runner: 2023-11-24T06:21:24.459 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-24T06:21:24.459 INFO:tasks.cephfs_test_runner:ERROR: test_volume_info_pending_subvol_deletions (tasks.cephfs.test_volumes.TestVolumes) 2023-11-24T06:21:24.459 INFO:tasks.cephfs_test_runner:Tests the pending_subvolume_deletions in 'fs volume info' command 2023-11-24T06:21:24.459 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T06:21:24.459 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-24T06:21:24.459 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs/test_volumes.py", line 568, in test_volume_info_pending_subvol_deletions 2023-11-24T06:21:24.460 INFO:tasks.cephfs_test_runner: self._do_subvolume_io(subvolname, number_of_files=3000, file_size=0) 2023-11-24T06:21:24.460 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs/test_volumes.py", line 278, in _do_subvolume_io 2023-11-24T06:21:24.460 INFO:tasks.cephfs_test_runner: filename = "{0}.{1}".format(TestVolumes.TEST_FILE_NAME_PREFIX, i) 2023-11-24T06:21:24.460 INFO:tasks.cephfs_test_runner:AttributeError: type object 'TestVolumes' has no attribute 'TEST_FILE_NAME_PREFIX' 2023-11-24T06:21:24.460 INFO:tasks.cephfs_test_runner: 2023-11-24T06:21:24.461 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_volume_info_pending_subvol_deletions (tasks.cephfs.test_volumes.TestVolumes) 2023-11-24T06:21:24.525 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0a6aa6a38f4a4a13bf3d798a7aa30b47 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_volume_info_pending_subvol_deletions (tasks.cephfs.test_volumes.TestVolumes) 2023-11-24T06:21:24.527 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-11-24T06:21:24.538 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-24T06:21:24.560 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-24T06:21:24.561 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:21:24.561 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:24.590 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:21:24.590 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-11-24T06:21:24.590 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:24.647 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284104k,nr_inodes=4071026,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16989 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260520k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:21:24.648 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:21:24.648 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:24.670 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:21:24.670 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-24T06:21:24.670 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:21:24.729 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17219 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:21:24.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:24.729 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:21:24.746 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T06:21:24.746 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T06:21:24.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:24.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T06:21:24.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:24.822 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:21:24.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:24.925 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-11-24T06:21:25.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:25.023 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-11-24T06:21:25.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:25.040 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-11-24T06:21:25.110 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2023-11-24T06:21:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:21:25.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:25.112 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:21:25.133 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T06:21:25.133 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T06:21:25.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:25.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T06:21:25.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:25.218 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:21:25.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:25.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2023-11-24T06:21:25.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:25.402 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2023-11-24T06:21:25.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:21:25.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2023-11-24T06:21:25.490 INFO:teuthology.orchestra.run.smithi123.stderr:Cannot find device "ceph-brx" 2023-11-24T06:21:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:21:25.492 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-24T06:21:25.502 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-24T06:21:25.502 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:25.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.621+0000 7f5d48f61700 1 Processor -- start 2023-11-24T06:21:25.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.621+0000 7f5d48f61700 1 -- start start 2023-11-24T06:21:25.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d48f61700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5d44158770 0x7f5d4406e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d48f61700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44071bd0 0x7f5d4406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d48f61700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d440b3000 con 0x7f5d4415b140 2023-11-24T06:21:25.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d48f61700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d440b32e0 con 0x7f5d44158770 2023-11-24T06:21:25.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d48f61700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d440b35c0 con 0x7f5d44071bd0 2023-11-24T06:21:25.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d42d9d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44071bd0 0x7f5d4406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d4259c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5d44158770 0x7f5d4406e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d42d9d700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44071bd0 0x7f5d4406e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:48520/0 (socket says 172.21.15.89:48520) 2023-11-24T06:21:25.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d42d9d700 1 -- 172.21.15.89:0/1273294570 learned_addr learned my addr 172.21.15.89:0/1273294570 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:25.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.622+0000 7f5d41d9b700 1 --1- 172.21.15.89:0/1273294570 >> v1:172.21.15.123:6789/0 conn(0x7f5d4415b140 0x7f5d4415e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.89:46966/0 (socket says 172.21.15.89:46966) 2023-11-24T06:21:25.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d42d9d700 1 -- 172.21.15.89:0/1273294570 >> v2:172.21.15.123:3300/0 conn(0x7f5d44158770 msgr2=0x7f5d4406e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:25.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d42d9d700 1 --2- 172.21.15.89:0/1273294570 >> v2:172.21.15.123:3300/0 conn(0x7f5d44158770 0x7f5d4406e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d42d9d700 1 -- 172.21.15.89:0/1273294570 >> v1:172.21.15.123:6789/0 conn(0x7f5d4415b140 legacy=0x7f5d4415e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d42d9d700 1 -- 172.21.15.89:0/1273294570 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d440b38a0 con 0x7f5d44071bd0 2023-11-24T06:21:25.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d4159a700 1 -- 172.21.15.89:0/1273294570 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614043074 0 0) 0x7f5d440b3000 con 0x7f5d4415b140 2023-11-24T06:21:25.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d4259c700 1 --2- 172.21.15.89:0/1273294570 >> v2:172.21.15.123:3300/0 conn(0x7f5d44158770 0x7f5d4406e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:21:25.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d42d9d700 1 --2- 172.21.15.89:0/1273294570 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44071bd0 0x7f5d4406e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00d150 tx=0x7f5d3c00d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93bc765c250dadad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:25.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d4159a700 1 -- 172.21.15.89:0/1273294570 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c010070 con 0x7f5d44071bd0 2023-11-24T06:21:25.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d4159a700 1 -- 172.21.15.89:0/1273294570 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c00dd10 con 0x7f5d44071bd0 2023-11-24T06:21:25.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.623+0000 7f5d4159a700 1 -- 172.21.15.89:0/1273294570 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00b310 con 0x7f5d44071bd0 2023-11-24T06:21:25.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.624+0000 7f5d48f61700 1 -- 172.21.15.89:0/1273294570 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44071bd0 msgr2=0x7f5d4406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.624+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1273294570 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44071bd0 0x7f5d4406e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00d150 tx=0x7f5d3c00d460 comp rx=0 tx=0).stop 2023-11-24T06:21:25.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.624+0000 7f5d48f61700 1 -- 172.21.15.89:0/1273294570 shutdown_connections 2023-11-24T06:21:25.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.624+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1273294570 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44071bd0 0x7f5d4406e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.624+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1273294570 >> v2:172.21.15.123:3300/0 conn(0x7f5d44158770 0x7f5d4406e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.624+0000 7f5d48f61700 1 -- 172.21.15.89:0/1273294570 >> 172.21.15.89:0/1273294570 conn(0x7f5d440b1740 msgr2=0x7f5d440b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:25.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.624+0000 7f5d48f61700 1 -- 172.21.15.89:0/1273294570 shutdown_connections 2023-11-24T06:21:25.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.624+0000 7f5d48f61700 1 -- 172.21.15.89:0/1273294570 wait complete. 2023-11-24T06:21:25.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.625+0000 7f5d48f61700 1 Processor -- start 2023-11-24T06:21:25.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.625+0000 7f5d48f61700 1 -- start start 2023-11-24T06:21:25.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d48f61700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d44071bd0 0x7f5d4414b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d48f61700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44158770 0x7f5d44151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d48f61700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d4415b140 0x7f5d4414a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d48f61700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4414af00 con 0x7f5d4415b140 2023-11-24T06:21:25.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d48f61700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d44155d60 con 0x7f5d44071bd0 2023-11-24T06:21:25.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d48f61700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d44156320 con 0x7f5d44158770 2023-11-24T06:21:25.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d4415b140 0x7f5d4414a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d4415b140 0x7f5d4414a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39650/0 (socket says 172.21.15.89:39650) 2023-11-24T06:21:25.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 -- 172.21.15.89:0/1175146700 learned_addr learned my addr 172.21.15.89:0/1175146700 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:25.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d4259c700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d44071bd0 0x7f5d4414b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d41d9b700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44158770 0x7f5d44151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 -- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44158770 msgr2=0x7f5d44151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44158770 0x7f5d44151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 -- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d44071bd0 msgr2=0x7f5d4414b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d44071bd0 0x7f5d4414b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 -- 172.21.15.89:0/1175146700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d380094a0 con 0x7f5d4415b140 2023-11-24T06:21:25.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d41d9b700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44158770 0x7f5d44151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:25.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.626+0000 7f5d42d9d700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d4415b140 0x7f5d4414a9c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00d7b0 tx=0x7f5d3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:25.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.627+0000 7f5d32ffd700 1 -- 172.21.15.89:0/1175146700 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00bbc0 con 0x7f5d4415b140 2023-11-24T06:21:25.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.627+0000 7f5d32ffd700 1 -- 172.21.15.89:0/1175146700 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c00b4c0 con 0x7f5d4415b140 2023-11-24T06:21:25.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.627+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c00bd40 con 0x7f5d4415b140 2023-11-24T06:21:25.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.627+0000 7f5d32ffd700 1 -- 172.21.15.89:0/1175146700 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c004040 con 0x7f5d4415b140 2023-11-24T06:21:25.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.627+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d440cf1a0 con 0x7f5d4415b140 2023-11-24T06:21:25.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.628+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d440751e0 con 0x7f5d4415b140 2023-11-24T06:21:25.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.628+0000 7f5d32ffd700 1 -- 172.21.15.89:0/1175146700 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f5d3c0041a0 con 0x7f5d4415b140 2023-11-24T06:21:25.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.628+0000 7f5d32ffd700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5d2406aff0 0x7f5d2406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.629+0000 7f5d32ffd700 1 -- 172.21.15.89:0/1175146700 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5d3c08cd20 con 0x7f5d4415b140 2023-11-24T06:21:25.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.629+0000 7f5d4259c700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5d2406aff0 0x7f5d2406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.629+0000 7f5d4259c700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5d2406aff0 0x7f5d2406d4b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5d440cc2d0 tx=0x7f5d38009210 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:25.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.633+0000 7f5d32ffd700 1 -- 172.21.15.89:0/1175146700 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d3c057f40 con 0x7f5d4415b140 2023-11-24T06:21:25.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.821+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d440b41d0 con 0x7f5d2406aff0 2023-11-24T06:21:25.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.823+0000 7f5d32ffd700 1 -- 172.21.15.89:0/1175146700 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50799 (secure 0 0 0) 0x7f5d38011b40 con 0x7f5d2406aff0 2023-11-24T06:21:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:25.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5d2406aff0 msgr2=0x7f5d2406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5d2406aff0 0x7f5d2406d4b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5d440cc2d0 tx=0x7f5d38009210 comp rx=0 tx=0).stop 2023-11-24T06:21:25.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d4415b140 msgr2=0x7f5d4414a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d4415b140 0x7f5d4414a9c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00d7b0 tx=0x7f5d3c004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 shutdown_connections 2023-11-24T06:21:25.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d4415b140 0x7f5d4414a9c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f5d2406aff0 0x7f5d2406d4b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d44071bd0 0x7f5d4414b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.825+0000 7f5d48f61700 1 --2- 172.21.15.89:0/1175146700 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d44158770 0x7f5d44151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.826+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 >> 172.21.15.89:0/1175146700 conn(0x7f5d440b1740 msgr2=0x7f5d4415d010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:25.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.826+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 shutdown_connections 2023-11-24T06:21:25.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.826+0000 7f5d48f61700 1 -- 172.21.15.89:0/1175146700 wait complete. 2023-11-24T06:21:25.833 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:25.840 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2023-11-24T06:21:25.750134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223608,"kb_used_data":7452,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751112,"statfs":{"total":773094113280,"available":772865138688,"internally_reserved":0,"allocated":7630848,"data_stored":3960007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26365,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-60,"num_write_kb":-46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000961"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":210,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.440877+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:58.440877+0000","last_peered":"2023-11-24T06:20:58.440877+0000","last_clean":"2023-11-24T06:20:58.440877+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:58.440877+0000","last_undegraded":"2023-11-24T06:20:58.440877+0000","last_fullsized":"2023-11-24T06:20:58.440877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":1396736,"data_stored":930812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.438},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.41599999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.35399999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58299999999999996}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65200000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.68799999999999994}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54400000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.629},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70399999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.67300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":839680,"data_stored":373268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.436},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.81699999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28356,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1372160,"data_stored":923986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.251},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.214}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.28699999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.317},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.33500000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57299999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.622},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.52700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":831488,"data_stored":372652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.38200000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.60099999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.69199999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.309}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.25600000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.29499999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.73999999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.622},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":794624,"data_stored":339769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.60199999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.436},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.68500000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.66400000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.76400000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.72299999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.629},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.495}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57799999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:25.842 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.959+0000 7fea0e8ca700 1 Processor -- start 2023-11-24T06:21:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.959+0000 7fea0e8ca700 1 -- start start 2023-11-24T06:21:25.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0e8ca700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fea0815c7d0 0x7fea0815cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0e8ca700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fea080d0790 0x7fea0815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0e8ca700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea0815fa50 con 0x7fea08062310 2023-11-24T06:21:25.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0e8ca700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0815fca0 con 0x7fea080d0790 2023-11-24T06:21:25.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0e8ca700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0815ff80 con 0x7fea0815c7d0 2023-11-24T06:21:25.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0ce67700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fea080d0790 0x7fea0815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea077fe700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fea08062310 0x7fea080d0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43158/0 (socket says 172.21.15.89:43158) 2023-11-24T06:21:25.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea077fe700 1 -- 172.21.15.89:0/2446102982 learned_addr learned my addr 172.21.15.89:0/2446102982 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:25.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea07fff700 1 --2- 172.21.15.89:0/2446102982 >> v2:172.21.15.123:3300/0 conn(0x7fea0815c7d0 0x7fea0815cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0ce67700 1 -- 172.21.15.89:0/2446102982 >> v2:172.21.15.123:3300/0 conn(0x7fea0815c7d0 msgr2=0x7fea0815cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0ce67700 1 --2- 172.21.15.89:0/2446102982 >> v2:172.21.15.123:3300/0 conn(0x7fea0815c7d0 0x7fea0815cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0ce67700 1 -- 172.21.15.89:0/2446102982 >> v1:172.21.15.89:6789/0 conn(0x7fea08062310 legacy=0x7fea080d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0ce67700 1 -- 172.21.15.89:0/2446102982 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08160260 con 0x7fea080d0790 2023-11-24T06:21:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea06ffd700 1 -- 172.21.15.89:0/2446102982 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831089657 0 0) 0x7fea0815fa50 con 0x7fea08062310 2023-11-24T06:21:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea0ce67700 1 --2- 172.21.15.89:0/2446102982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea080d0790 0x7fea0815d7d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fea080b63e0 tx=0x7fea0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19575c9507cec18 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea06ffd700 1 -- 172.21.15.89:0/2446102982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0000e070 con 0x7fea080d0790 2023-11-24T06:21:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea06ffd700 1 -- 172.21.15.89:0/2446102982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0000bb20 con 0x7fea080d0790 2023-11-24T06:21:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.960+0000 7fea06ffd700 1 -- 172.21.15.89:0/2446102982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00012430 con 0x7fea080d0790 2023-11-24T06:21:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.961+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2446102982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea080d0790 msgr2=0x7fea0815d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.961+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2446102982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea080d0790 0x7fea0815d7d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fea080b63e0 tx=0x7fea0000b330 comp rx=0 tx=0).stop 2023-11-24T06:21:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.961+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2446102982 shutdown_connections 2023-11-24T06:21:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.961+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2446102982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea080d0790 0x7fea0815d7d0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.961+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2446102982 >> v2:172.21.15.123:3300/0 conn(0x7fea0815c7d0 0x7fea0815cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.961+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2446102982 >> 172.21.15.89:0/2446102982 conn(0x7fea080b5640 msgr2=0x7fea080b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.961+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2446102982 shutdown_connections 2023-11-24T06:21:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.961+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2446102982 wait complete. 2023-11-24T06:21:25.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.962+0000 7fea0e8ca700 1 Processor -- start 2023-11-24T06:21:25.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.962+0000 7fea0e8ca700 1 -- start start 2023-11-24T06:21:25.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0e8ca700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea08062310 0x7fea081510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0e8ca700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea080d0790 0x7fea0809d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0e8ca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea0815c7d0 0x7fea0809d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0e8ca700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0814c290 con 0x7fea0815c7d0 2023-11-24T06:21:25.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0e8ca700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0814c530 con 0x7fea080d0790 2023-11-24T06:21:25.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0e8ca700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0814c810 con 0x7fea08062310 2023-11-24T06:21:25.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0ce67700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea0815c7d0 0x7fea0809d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0ce67700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea0815c7d0 0x7fea0809d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39664/0 (socket says 172.21.15.89:39664) 2023-11-24T06:21:25.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0ce67700 1 -- 172.21.15.89:0/2478841532 learned_addr learned my addr 172.21.15.89:0/2478841532 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:25.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea077fe700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea080d0790 0x7fea0809d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea07fff700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea08062310 0x7fea081510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0ce67700 1 -- 172.21.15.89:0/2478841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea08062310 msgr2=0x7fea081510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0ce67700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea08062310 0x7fea081510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0ce67700 1 -- 172.21.15.89:0/2478841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea080d0790 msgr2=0x7fea0809d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:25.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.963+0000 7fea0ce67700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea080d0790 0x7fea0809d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:25.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.964+0000 7fea0ce67700 1 -- 172.21.15.89:0/2478841532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00009a70 con 0x7fea0815c7d0 2023-11-24T06:21:25.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.964+0000 7fea0ce67700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea0815c7d0 0x7fea0809d5d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fea00014040 tx=0x7fea00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:25.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.964+0000 7fea04ff9700 1 -- 172.21.15.89:0/2478841532 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00012430 con 0x7fea0815c7d0 2023-11-24T06:21:25.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.964+0000 7fea04ff9700 1 -- 172.21.15.89:0/2478841532 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea00020070 con 0x7fea0815c7d0 2023-11-24T06:21:25.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.964+0000 7fea04ff9700 1 -- 172.21.15.89:0/2478841532 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00004280 con 0x7fea0815c7d0 2023-11-24T06:21:25.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.964+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0814caf0 con 0x7fea0815c7d0 2023-11-24T06:21:25.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.964+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea08151600 con 0x7fea0815c7d0 2023-11-24T06:21:25.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.965+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0814cce0 con 0x7fea0815c7d0 2023-11-24T06:21:25.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.965+0000 7fea04ff9700 1 -- 172.21.15.89:0/2478841532 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fea000043e0 con 0x7fea0815c7d0 2023-11-24T06:21:25.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.965+0000 7fea04ff9700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9f406aff0 0x7fe9f406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:25.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.966+0000 7fea04ff9700 1 -- 172.21.15.89:0/2478841532 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fea00022070 con 0x7fea0815c7d0 2023-11-24T06:21:25.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.966+0000 7fea07fff700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9f406aff0 0x7fe9f406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:25.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.966+0000 7fea07fff700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9f406aff0 0x7fe9f406d4b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc009b70 tx=0x7fe9fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:25.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:25.970+0000 7fea04ff9700 1 -- 172.21.15.89:0/2478841532 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea00056440 con 0x7fea0815c7d0 2023-11-24T06:21:26.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.157+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea08156150 con 0x7fe9f406aff0 2023-11-24T06:21:26.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.159+0000 7fea04ff9700 1 -- 172.21.15.89:0/2478841532 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50799 (secure 0 0 0) 0x7fea08156150 con 0x7fe9f406aff0 2023-11-24T06:21:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:26.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9f406aff0 msgr2=0x7fe9f406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9f406aff0 0x7fe9f406d4b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc009b70 tx=0x7fe9fc008040 comp rx=0 tx=0).stop 2023-11-24T06:21:26.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea0815c7d0 msgr2=0x7fea0809d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea0815c7d0 0x7fea0809d5d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fea00014040 tx=0x7fea00004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 shutdown_connections 2023-11-24T06:21:26.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea0815c7d0 0x7fea0809d5d0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea08062310 0x7fea081510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe9f406aff0 0x7fe9f406d4b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 --2- 172.21.15.89:0/2478841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea080d0790 0x7fea0809d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 >> 172.21.15.89:0/2478841532 conn(0x7fea080b5640 msgr2=0x7fea0809b520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:26.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 shutdown_connections 2023-11-24T06:21:26.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.161+0000 7fea0e8ca700 1 -- 172.21.15.89:0/2478841532 wait complete. 2023-11-24T06:21:26.168 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:26.176 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2023-11-24T06:21:25.750134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223608,"kb_used_data":7452,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751112,"statfs":{"total":773094113280,"available":772865138688,"internally_reserved":0,"allocated":7630848,"data_stored":3960007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26365,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-60,"num_write_kb":-46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000961"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":210,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.440877+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:58.440877+0000","last_peered":"2023-11-24T06:20:58.440877+0000","last_clean":"2023-11-24T06:20:58.440877+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:58.440877+0000","last_undegraded":"2023-11-24T06:20:58.440877+0000","last_fullsized":"2023-11-24T06:20:58.440877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":1396736,"data_stored":930812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.438},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.41599999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.35399999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58299999999999996}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65200000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.68799999999999994}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54400000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.629},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70399999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.67300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":839680,"data_stored":373268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.436},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.81699999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28356,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1372160,"data_stored":923986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.251},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.214}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.28699999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.317},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.33500000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57299999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.622},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.52700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":831488,"data_stored":372652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.38200000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.60099999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.69199999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.309}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.25600000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.29499999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.73999999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.622},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":794624,"data_stored":339769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.60199999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.436},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.68500000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.66400000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.76400000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.72299999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.629},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.495}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57799999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:26.178 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-24T06:21:26.178 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:26.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.295+0000 7f86ff705700 1 Processor -- start 2023-11-24T06:21:26.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.295+0000 7f86ff705700 1 -- start start 2023-11-24T06:21:26.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86ff705700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86f81446a0 0x7f86f80b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86ff705700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f86f806c710 0x7f86f80b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86ff705700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f80af950 con 0x7f86f80adf00 2023-11-24T06:21:26.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86ff705700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f8144a80 con 0x7f86f81446a0 2023-11-24T06:21:26.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86ff705700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f80afc30 con 0x7f86f806c710 2023-11-24T06:21:26.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86fd4a1700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86f81446a0 0x7f86f80b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86fcca0700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f86f80adf00 0x7f86f806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43172/0 (socket says 172.21.15.89:43172) 2023-11-24T06:21:26.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86fcca0700 1 -- 172.21.15.89:0/1521900870 learned_addr learned my addr 172.21.15.89:0/1521900870 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:26.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.296+0000 7f86fd4a1700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86f81446a0 0x7f86f80b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39668/0 (socket says 172.21.15.89:39668) 2023-11-24T06:21:26.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86fdca2700 1 --2- 172.21.15.89:0/1521900870 >> v2:172.21.15.123:3300/0 conn(0x7f86f806c710 0x7f86f80b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86fd4a1700 1 -- 172.21.15.89:0/1521900870 >> v2:172.21.15.123:3300/0 conn(0x7f86f806c710 msgr2=0x7f86f80b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:26.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86fd4a1700 1 --2- 172.21.15.89:0/1521900870 >> v2:172.21.15.123:3300/0 conn(0x7f86f806c710 0x7f86f80b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86fd4a1700 1 -- 172.21.15.89:0/1521900870 >> v1:172.21.15.89:6789/0 conn(0x7f86f80adf00 legacy=0x7f86f806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86fd4a1700 1 -- 172.21.15.89:0/1521900870 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f80aff10 con 0x7f86f81446a0 2023-11-24T06:21:26.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86effff700 1 -- 172.21.15.89:0/1521900870 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747084061 0 0) 0x7f86f80af950 con 0x7f86f80adf00 2023-11-24T06:21:26.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86fdca2700 1 --2- 172.21.15.89:0/1521900870 >> v2:172.21.15.123:3300/0 conn(0x7f86f806c710 0x7f86f80b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:26.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86fd4a1700 1 --2- 172.21.15.89:0/1521900870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f81446a0 0x7f86f80b5ae0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f86f80aedf0 tx=0x7f86e8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bc08dd08e5ffb5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:26.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86effff700 1 -- 172.21.15.89:0/1521900870 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8011830 con 0x7f86f81446a0 2023-11-24T06:21:26.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86effff700 1 -- 172.21.15.89:0/1521900870 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e8011990 con 0x7f86f81446a0 2023-11-24T06:21:26.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.297+0000 7f86effff700 1 -- 172.21.15.89:0/1521900870 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8011d20 con 0x7f86f81446a0 2023-11-24T06:21:26.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.298+0000 7f86ff705700 1 -- 172.21.15.89:0/1521900870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f81446a0 msgr2=0x7f86f80b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.298+0000 7f86ff705700 1 --2- 172.21.15.89:0/1521900870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f81446a0 0x7f86f80b5ae0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f86f80aedf0 tx=0x7f86e8009b20 comp rx=0 tx=0).stop 2023-11-24T06:21:26.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.298+0000 7f86ff705700 1 -- 172.21.15.89:0/1521900870 shutdown_connections 2023-11-24T06:21:26.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.298+0000 7f86ff705700 1 --2- 172.21.15.89:0/1521900870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f81446a0 0x7f86f80b5ae0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.298+0000 7f86ff705700 1 --2- 172.21.15.89:0/1521900870 >> v2:172.21.15.123:3300/0 conn(0x7f86f806c710 0x7f86f80b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.298+0000 7f86ff705700 1 -- 172.21.15.89:0/1521900870 >> 172.21.15.89:0/1521900870 conn(0x7f86f8071b90 msgr2=0x7f86f8071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:26.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.298+0000 7f86ff705700 1 -- 172.21.15.89:0/1521900870 shutdown_connections 2023-11-24T06:21:26.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.298+0000 7f86ff705700 1 -- 172.21.15.89:0/1521900870 wait complete. 2023-11-24T06:21:26.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.299+0000 7f86ff705700 1 Processor -- start 2023-11-24T06:21:26.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.299+0000 7f86ff705700 1 -- start start 2023-11-24T06:21:26.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86ff705700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f806c710 0x7f86f80d0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86ff705700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80adf00 0x7f86f80ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86ff705700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 0x7f86f80ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86ff705700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f80cf660 con 0x7f86f806c710 2023-11-24T06:21:26.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86ff705700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f8167b40 con 0x7f86f80adf00 2023-11-24T06:21:26.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86ff705700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f81647f0 con 0x7f86f81446a0 2023-11-24T06:21:26.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f806c710 0x7f86f80d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f806c710 0x7f86f80d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39672/0 (socket says 172.21.15.89:39672) 2023-11-24T06:21:26.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 -- 172.21.15.89:0/3584575751 learned_addr learned my addr 172.21.15.89:0/3584575751 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:26.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fcca0700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80adf00 0x7f86f80ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fdca2700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 0x7f86f80ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 -- 172.21.15.89:0/3584575751 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 msgr2=0x7f86f80ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 0x7f86f80ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 -- 172.21.15.89:0/3584575751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80adf00 msgr2=0x7f86f80ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80adf00 0x7f86f80ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 -- 172.21.15.89:0/3584575751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e80094a0 con 0x7f86f806c710 2023-11-24T06:21:26.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86fd4a1700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f806c710 0x7f86f80d0070 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009fd0 tx=0x7f86e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:26.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86edffb700 1 -- 172.21.15.89:0/3584575751 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e80038a0 con 0x7f86f806c710 2023-11-24T06:21:26.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86edffb700 1 -- 172.21.15.89:0/3584575751 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e801e070 con 0x7f86f806c710 2023-11-24T06:21:26.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f8164ad0 con 0x7f86f806c710 2023-11-24T06:21:26.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.300+0000 7f86edffb700 1 -- 172.21.15.89:0/3584575751 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8003c00 con 0x7f86f806c710 2023-11-24T06:21:26.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.301+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f8165090 con 0x7f86f806c710 2023-11-24T06:21:26.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.302+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f80753f0 con 0x7f86f806c710 2023-11-24T06:21:26.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.302+0000 7f86edffb700 1 -- 172.21.15.89:0/3584575751 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f86e802d450 con 0x7f86f806c710 2023-11-24T06:21:26.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.302+0000 7f86edffb700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86dc06aff0 0x7f86dc06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.303+0000 7f86edffb700 1 -- 172.21.15.89:0/3584575751 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f86e808a4b0 con 0x7f86f806c710 2023-11-24T06:21:26.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.303+0000 7f86fcca0700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86dc06aff0 0x7f86dc06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.303+0000 7f86fcca0700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86dc06aff0 0x7f86dc06d4b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f86f81f72e0 tx=0x7f86f4007c00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:26.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.307+0000 7f86edffb700 1 -- 172.21.15.89:0/3584575751 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86e80557e0 con 0x7f86f806c710 2023-11-24T06:21:26.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.494+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f86f80cccd0 con 0x7f86dc06aff0 2023-11-24T06:21:26.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.496+0000 7f86edffb700 1 -- 172.21.15.89:0/3584575751 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50799 (secure 0 0 0) 0x7f86f80cccd0 con 0x7f86dc06aff0 2023-11-24T06:21:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:26.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86dc06aff0 msgr2=0x7f86dc06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86dc06aff0 0x7f86dc06d4b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f86f81f72e0 tx=0x7f86f4007c00 comp rx=0 tx=0).stop 2023-11-24T06:21:26.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f806c710 msgr2=0x7f86f80d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f806c710 0x7f86f80d0070 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009fd0 tx=0x7f86e8004750 comp rx=0 tx=0).stop 2023-11-24T06:21:26.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 shutdown_connections 2023-11-24T06:21:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 0x7f86f80ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f806c710 0x7f86f80d0070 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f86dc06aff0 0x7f86dc06d4b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 --2- 172.21.15.89:0/3584575751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80adf00 0x7f86f80ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 >> 172.21.15.89:0/3584575751 conn(0x7f86f8071b90 msgr2=0x7f86f80b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 shutdown_connections 2023-11-24T06:21:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.498+0000 7f86ff705700 1 -- 172.21.15.89:0/3584575751 wait complete. 2023-11-24T06:21:26.506 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:26.513 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2023-11-24T06:21:25.750134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223608,"kb_used_data":7452,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751112,"statfs":{"total":773094113280,"available":772865138688,"internally_reserved":0,"allocated":7630848,"data_stored":3960007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26365,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-60,"num_write_kb":-46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000961"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":210,"reported_epoch":64,"state":"active+clean","last_fresh":"2023-11-24T06:20:58.440877+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:20:58.440877+0000","last_peered":"2023-11-24T06:20:58.440877+0000","last_clean":"2023-11-24T06:20:58.440877+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:20:58.440877+0000","last_undegraded":"2023-11-24T06:20:58.440877+0000","last_fullsized":"2023-11-24T06:20:58.440877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":1396736,"data_stored":930812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.438},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.41599999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.35399999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58299999999999996}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65200000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.68799999999999994}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54400000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.629},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70399999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.67300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":839680,"data_stored":373268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.436},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.81699999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28356,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1372160,"data_stored":923986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.251},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.214}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.28699999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.317},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.33500000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57299999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.622},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.52700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":831488,"data_stored":372652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.38200000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.60099999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.69199999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.309}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.25600000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.29499999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.73999999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.622},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":794624,"data_stored":339769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.60199999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.436},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.68500000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.66400000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.76400000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.72299999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":339918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.629},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.495}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57799999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:26.515 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:21:26.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.632+0000 7fc902c19700 1 Processor -- start 2023-11-24T06:21:26.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.632+0000 7fc902c19700 1 -- start start 2023-11-24T06:21:26.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.632+0000 7fc902c19700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc1446a0 0x7fc8fc0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.632+0000 7fc902c19700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8fc06c710 0x7fc8fc0b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.632+0000 7fc902c19700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc0af950 con 0x7fc8fc0adf00 2023-11-24T06:21:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.632+0000 7fc902c19700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8fc144a80 con 0x7fc8fc06c710 2023-11-24T06:21:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.632+0000 7fc902c19700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0afc30 con 0x7fc8fc1446a0 2023-11-24T06:21:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc8fbfff700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc8fc0adf00 0x7fc8fc06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43184/0 (socket says 172.21.15.89:43184) 2023-11-24T06:21:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc8fbfff700 1 -- 172.21.15.89:0/555542449 learned_addr learned my addr 172.21.15.89:0/555542449 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc9011b6700 1 --2- 172.21.15.89:0/555542449 >> v2:172.21.15.89:3300/0 conn(0x7fc8fc06c710 0x7fc8fc0b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc8fb7fe700 1 -- 172.21.15.89:0/555542449 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672339051 0 0) 0x7fc8fc0af950 con 0x7fc8fc0adf00 2023-11-24T06:21:26.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc9009b5700 1 --2- 172.21.15.89:0/555542449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc1446a0 0x7fc8fc0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc8fb7fe700 1 -- 172.21.15.89:0/555542449 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec0035b0 con 0x7fc8fc0adf00 2023-11-24T06:21:26.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc9011b6700 1 -- 172.21.15.89:0/555542449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc1446a0 msgr2=0x7fc8fc0b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:26.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc9011b6700 1 --2- 172.21.15.89:0/555542449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc1446a0 0x7fc8fc0b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc9011b6700 1 -- 172.21.15.89:0/555542449 >> v1:172.21.15.89:6789/0 conn(0x7fc8fc0adf00 legacy=0x7fc8fc06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc9011b6700 1 -- 172.21.15.89:0/555542449 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc0aff10 con 0x7fc8fc06c710 2023-11-24T06:21:26.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc9011b6700 1 --2- 172.21.15.89:0/555542449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc06c710 0x7fc8fc0b3d40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009d90 tx=0x7fc8f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe52eab8f2185b7c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:26.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc8fb7fe700 1 -- 172.21.15.89:0/555542449 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f400e070 con 0x7fc8fc06c710 2023-11-24T06:21:26.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc8fb7fe700 1 -- 172.21.15.89:0/555542449 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f400bb20 con 0x7fc8fc06c710 2023-11-24T06:21:26.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.633+0000 7fc8fb7fe700 1 -- 172.21.15.89:0/555542449 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4012430 con 0x7fc8fc06c710 2023-11-24T06:21:26.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.634+0000 7fc902c19700 1 -- 172.21.15.89:0/555542449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc06c710 msgr2=0x7fc8fc0b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.634+0000 7fc902c19700 1 --2- 172.21.15.89:0/555542449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc06c710 0x7fc8fc0b3d40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009d90 tx=0x7fc8f400b330 comp rx=0 tx=0).stop 2023-11-24T06:21:26.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.634+0000 7fc902c19700 1 -- 172.21.15.89:0/555542449 shutdown_connections 2023-11-24T06:21:26.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.634+0000 7fc902c19700 1 --2- 172.21.15.89:0/555542449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc06c710 0x7fc8fc0b3d40 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.634+0000 7fc902c19700 1 --2- 172.21.15.89:0/555542449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc1446a0 0x7fc8fc0b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.634+0000 7fc902c19700 1 -- 172.21.15.89:0/555542449 >> 172.21.15.89:0/555542449 conn(0x7fc8fc071b90 msgr2=0x7fc8fc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:26.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.634+0000 7fc902c19700 1 -- 172.21.15.89:0/555542449 shutdown_connections 2023-11-24T06:21:26.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.634+0000 7fc902c19700 1 -- 172.21.15.89:0/555542449 wait complete. 2023-11-24T06:21:26.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.635+0000 7fc902c19700 1 Processor -- start 2023-11-24T06:21:26.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.635+0000 7fc902c19700 1 -- start start 2023-11-24T06:21:26.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc902c19700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc06c710 0x7fc8fc137580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc902c19700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8fc0adf00 0x7fc8fc143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc902c19700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc1446a0 0x7fc8fc143860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc902c19700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc15c870 con 0x7fc8fc1446a0 2023-11-24T06:21:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc902c19700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc15cb50 con 0x7fc8fc0adf00 2023-11-24T06:21:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc902c19700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc15ce30 con 0x7fc8fc06c710 2023-11-24T06:21:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9011b6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc1446a0 0x7fc8fc143860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9011b6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc1446a0 0x7fc8fc143860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39690/0 (socket says 172.21.15.89:39690) 2023-11-24T06:21:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9011b6700 1 -- 172.21.15.89:0/1250760693 learned_addr learned my addr 172.21.15.89:0/1250760693 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9009b5700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc06c710 0x7fc8fc137580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc8fbfff700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8fc0adf00 0x7fc8fc143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9011b6700 1 -- 172.21.15.89:0/1250760693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc06c710 msgr2=0x7fc8fc137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9011b6700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc06c710 0x7fc8fc137580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9011b6700 1 -- 172.21.15.89:0/1250760693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8fc0adf00 msgr2=0x7fc8fc143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9011b6700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8fc0adf00 0x7fc8fc143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.636+0000 7fc9011b6700 1 -- 172.21.15.89:0/1250760693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f4009a70 con 0x7fc8fc1446a0 2023-11-24T06:21:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.637+0000 7fc9011b6700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc1446a0 0x7fc8fc143860 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4014040 tx=0x7fc8f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.637+0000 7fc8f97fa700 1 -- 172.21.15.89:0/1250760693 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4012430 con 0x7fc8fc1446a0 2023-11-24T06:21:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.637+0000 7fc8f97fa700 1 -- 172.21.15.89:0/1250760693 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4020070 con 0x7fc8fc1446a0 2023-11-24T06:21:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.637+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc15d110 con 0x7fc8fc1446a0 2023-11-24T06:21:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.637+0000 7fc8f97fa700 1 -- 172.21.15.89:0/1250760693 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4003ea0 con 0x7fc8fc1446a0 2023-11-24T06:21:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.637+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc0d0800 con 0x7fc8fc1446a0 2023-11-24T06:21:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.638+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8fc0753f0 con 0x7fc8fc1446a0 2023-11-24T06:21:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.638+0000 7fc8f97fa700 1 -- 172.21.15.89:0/1250760693 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc8f4004280 con 0x7fc8fc1446a0 2023-11-24T06:21:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.639+0000 7fc8f97fa700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8e006aff0 0x7fc8e006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.639+0000 7fc8f97fa700 1 -- 172.21.15.89:0/1250760693 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc8f4022070 con 0x7fc8fc1446a0 2023-11-24T06:21:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.639+0000 7fc9009b5700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8e006aff0 0x7fc8e006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.639+0000 7fc9009b5700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8e006aff0 0x7fc8e006d4b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0ccef0 tx=0x7fc8f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.643+0000 7fc8f97fa700 1 -- 172.21.15.89:0/1250760693 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc8f4056810 con 0x7fc8fc1446a0 2023-11-24T06:21:26.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.834+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8fc15d5d0 con 0x7fc8fc1446a0 2023-11-24T06:21:26.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.835+0000 7fc8f97fa700 1 -- 172.21.15.89:0/1250760693 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+12953 (secure 0 0 0) 0x7fc8f4059e30 con 0x7fc8fc1446a0 2023-11-24T06:21:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:26.838 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":67,"fsid":"be2488f4-68a7-42cc-93d1-ae9c6921fb48","created":"2023-11-24T06:18:24.133686+0000","modified":"2023-11-24T06:21:23.432457+0000","last_up_change":"2023-11-24T06:18:35.752888+0000","last_in_change":"2023-11-24T06:18:26.628120+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:18:37.761810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3933529-39bc-4e58-a042-b9028c56f953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6801","nonce":1704280696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6803","nonce":1704280696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6807","nonce":1704280696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1704280696},{"type":"v1","addr":"172.21.15.89:6805","nonce":1704280696}]},"public_addr":"172.21.15.89:6801/1704280696","cluster_addr":"172.21.15.89:6803/1704280696","heartbeat_back_addr":"172.21.15.89:6807/1704280696","heartbeat_front_addr":"172.21.15.89:6805/1704280696","state":["exists","up"]},{"osd":1,"uuid":"97e78a03-85d2-414c-8968-f999dd8e1126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6809","nonce":4291702656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6811","nonce":4291702656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6815","nonce":4291702656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":4291702656},{"type":"v1","addr":"172.21.15.89:6813","nonce":4291702656}]},"public_addr":"172.21.15.89:6809/4291702656","cluster_addr":"172.21.15.89:6811/4291702656","heartbeat_back_addr":"172.21.15.89:6815/4291702656","heartbeat_front_addr":"172.21.15.89:6813/4291702656","state":["exists","up"]},{"osd":2,"uuid":"b7abce2a-3e50-4e84-b63c-8758106b1da9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6817","nonce":341459167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6819","nonce":341459167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6823","nonce":341459167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":341459167},{"type":"v1","addr":"172.21.15.89:6821","nonce":341459167}]},"public_addr":"172.21.15.89:6817/341459167","cluster_addr":"172.21.15.89:6819/341459167","heartbeat_back_addr":"172.21.15.89:6823/341459167","heartbeat_front_addr":"172.21.15.89:6821/341459167","state":["exists","up"]},{"osd":3,"uuid":"7286d2e5-69a1-4cae-9893-0d8f496e78bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6825","nonce":1593118452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6827","nonce":1593118452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6831","nonce":1593118452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1593118452},{"type":"v1","addr":"172.21.15.89:6829","nonce":1593118452}]},"public_addr":"172.21.15.89:6825/1593118452","cluster_addr":"172.21.15.89:6827/1593118452","heartbeat_back_addr":"172.21.15.89:6831/1593118452","heartbeat_front_addr":"172.21.15.89:6829/1593118452","state":["exists","up"]},{"osd":4,"uuid":"42ec5b7e-0658-47eb-8f04-d6a205b93ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6817","nonce":1261947525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6819","nonce":1261947525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6823","nonce":1261947525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1261947525},{"type":"v1","addr":"172.21.15.123:6821","nonce":1261947525}]},"public_addr":"172.21.15.123:6817/1261947525","cluster_addr":"172.21.15.123:6819/1261947525","heartbeat_back_addr":"172.21.15.123:6823/1261947525","heartbeat_front_addr":"172.21.15.123:6821/1261947525","state":["exists","up"]},{"osd":5,"uuid":"f62c4eff-c5f6-4f27-9c1c-d87dad75c6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6809","nonce":410207526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6811","nonce":410207526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6815","nonce":410207526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":410207526},{"type":"v1","addr":"172.21.15.123:6813","nonce":410207526}]},"public_addr":"172.21.15.123:6809/410207526","cluster_addr":"172.21.15.123:6811/410207526","heartbeat_back_addr":"172.21.15.123:6815/410207526","heartbeat_front_addr":"172.21.15.123:6813/410207526","state":["exists","up"]},{"osd":6,"uuid":"0317ea02-1e74-45a8-8734-eaa2f69e85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6825","nonce":668316489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6827","nonce":668316489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6831","nonce":668316489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":668316489},{"type":"v1","addr":"172.21.15.123:6829","nonce":668316489}]},"public_addr":"172.21.15.123:6825/668316489","cluster_addr":"172.21.15.123:6827/668316489","heartbeat_back_addr":"172.21.15.123:6831/668316489","heartbeat_front_addr":"172.21.15.123:6829/668316489","state":["exists","up"]},{"osd":7,"uuid":"1f14683e-7e01-4616-9c01-3dafbbda1f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6801","nonce":751799151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6803","nonce":751799151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6807","nonce":751799151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":751799151},{"type":"v1","addr":"172.21.15.123:6805","nonce":751799151}]},"public_addr":"172.21.15.123:6801/751799151","cluster_addr":"172.21.15.123:6803/751799151","heartbeat_back_addr":"172.21.15.123:6807/751799151","heartbeat_front_addr":"172.21.15.123:6805/751799151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.856136+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.819245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.858898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.813672+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.880167+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.796492+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.859078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:18:31.823685+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/2963752713":"2023-11-25T06:21:20.410781+0000","172.21.15.123:6833/2963752713":"2023-11-25T06:21:20.410781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:21:26.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8e006aff0 msgr2=0x7fc8e006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8e006aff0 0x7fc8e006d4b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0ccef0 tx=0x7fc8f0008040 comp rx=0 tx=0).stop 2023-11-24T06:21:26.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc1446a0 msgr2=0x7fc8fc143860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:26.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc1446a0 0x7fc8fc143860 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4014040 tx=0x7fc8f4004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 shutdown_connections 2023-11-24T06:21:26.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc1446a0 0x7fc8fc143860 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8fc06c710 0x7fc8fc137580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc8e006aff0 0x7fc8e006d4b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 --2- 172.21.15.89:0/1250760693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8fc0adf00 0x7fc8fc143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:26.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 >> 172.21.15.89:0/1250760693 conn(0x7fc8fc071b90 msgr2=0x7fc8fc13df20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:26.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 shutdown_connections 2023-11-24T06:21:26.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:26.837+0000 7fc902c19700 1 -- 172.21.15.89:0/1250760693 wait complete. 2023-11-24T06:21:27.852 INFO:tasks.ceph:Scrubbing osd.0 2023-11-24T06:21:27.853 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:21:27.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.982+0000 7f93fd645700 1 Processor -- start 2023-11-24T06:21:27.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.982+0000 7f93fd645700 1 -- start start 2023-11-24T06:21:27.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93fd645700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f93f8073f90 0x7f93f806f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:27.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93fd645700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93f80b5850 0x7f93f806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:27.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93fd645700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f8074370 0x7f93f806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:27.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93fd645700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f81470a0 con 0x7f93f80b5850 2023-11-24T06:21:27.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93fd645700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f80b3b50 con 0x7f93f8073f90 2023-11-24T06:21:27.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93fd645700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80b43b0 con 0x7f93f8074370 2023-11-24T06:21:27.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93f67fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93f80b5850 0x7f93f806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:27.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93f67fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93f80b5850 0x7f93f806e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39694/0 (socket says 172.21.15.89:39694) 2023-11-24T06:21:27.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93f67fc700 1 -- 172.21.15.89:0/1838452186 learned_addr learned my addr 172.21.15.89:0/1838452186 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:27.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93f77fe700 1 --2- 172.21.15.89:0/1838452186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f8074370 0x7f93f806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:27.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.983+0000 7f93f6ffd700 1 --2- 172.21.15.89:0/1838452186 >> v2:172.21.15.123:3300/0 conn(0x7f93f8073f90 0x7f93f806f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:27.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f67fc700 1 -- 172.21.15.89:0/1838452186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f8074370 msgr2=0x7f93f806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:27.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f67fc700 1 --2- 172.21.15.89:0/1838452186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f8074370 0x7f93f806aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:27.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f67fc700 1 -- 172.21.15.89:0/1838452186 >> v2:172.21.15.123:3300/0 conn(0x7f93f8073f90 msgr2=0x7f93f806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:27.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f67fc700 1 --2- 172.21.15.89:0/1838452186 >> v2:172.21.15.123:3300/0 conn(0x7f93f8073f90 0x7f93f806f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:27.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f67fc700 1 -- 172.21.15.89:0/1838452186 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80b5d30 con 0x7f93f80b5850 2023-11-24T06:21:27.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f67fc700 1 --2- 172.21.15.89:0/1838452186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f80b5850 0x7f93f806e9b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f93f8073410 tx=0x7f93e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b549ea7c3a5e44a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:27.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f5ffb700 1 -- 172.21.15.89:0/1838452186 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e8011830 con 0x7f93f80b5850 2023-11-24T06:21:27.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f5ffb700 1 -- 172.21.15.89:0/1838452186 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e8011990 con 0x7f93f80b5850 2023-11-24T06:21:27.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.984+0000 7f93f5ffb700 1 -- 172.21.15.89:0/1838452186 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e8011d20 con 0x7f93f80b5850 2023-11-24T06:21:27.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 -- 172.21.15.89:0/1838452186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f80b5850 msgr2=0x7f93f806e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:27.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 --2- 172.21.15.89:0/1838452186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f80b5850 0x7f93f806e9b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f93f8073410 tx=0x7f93e8009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:27.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 -- 172.21.15.89:0/1838452186 shutdown_connections 2023-11-24T06:21:27.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 --2- 172.21.15.89:0/1838452186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f8074370 0x7f93f806aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:27.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 --2- 172.21.15.89:0/1838452186 >> v2:172.21.15.123:3300/0 conn(0x7f93f8073f90 0x7f93f806f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:27.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 --2- 172.21.15.89:0/1838452186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f80b5850 0x7f93f806e9b0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:27.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 -- 172.21.15.89:0/1838452186 >> 172.21.15.89:0/1838452186 conn(0x7f93f80d0b40 msgr2=0x7f93f80d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:27.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 -- 172.21.15.89:0/1838452186 shutdown_connections 2023-11-24T06:21:27.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.985+0000 7f93fd645700 1 -- 172.21.15.89:0/1838452186 wait complete. 2023-11-24T06:21:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.986+0000 7f93fd645700 1 Processor -- start 2023-11-24T06:21:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.986+0000 7f93fd645700 1 -- start start 2023-11-24T06:21:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93fd645700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8073f90 0x7f93f815d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93fd645700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f8074370 0x7f93f8161c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93fd645700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f80b5850 0x7f93f8144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93fd645700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f81f7ba0 con 0x7f93f8073f90 2023-11-24T06:21:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93fd645700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f81f7670 con 0x7f93f8074370 2023-11-24T06:21:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93fd645700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f806ba70 con 0x7f93f80b5850 2023-11-24T06:21:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8073f90 0x7f93f815d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8073f90 0x7f93f815d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39700/0 (socket says 172.21.15.89:39700) 2023-11-24T06:21:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 -- 172.21.15.89:0/3264632189 learned_addr learned my addr 172.21.15.89:0/3264632189 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f67fc700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f8074370 0x7f93f8161c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 -- 172.21.15.89:0/3264632189 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f80b5850 msgr2=0x7f93f8144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f77fe700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f80b5850 0x7f93f8144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f80b5850 0x7f93f8144070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 -- 172.21.15.89:0/3264632189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f8074370 msgr2=0x7f93f8161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f8074370 0x7f93f8161c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 -- 172.21.15.89:0/3264632189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e8009550 con 0x7f93f8073f90 2023-11-24T06:21:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f67fc700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f8074370 0x7f93f8161c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f77fe700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f80b5850 0x7f93f8144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93f6ffd700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8073f90 0x7f93f815d9f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00e9b0 tx=0x7f93ec00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ec009710 con 0x7f93f8073f90 2023-11-24T06:21:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ec00c850 con 0x7f93f8073f90 2023-11-24T06:21:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f81f78c0 con 0x7f93f8073f90 2023-11-24T06:21:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ec00cbc0 con 0x7f93f8073f90 2023-11-24T06:21:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.987+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f80c6920 con 0x7f93f8073f90 2023-11-24T06:21:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.988+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f93f80b40d0 con 0x7f93f8073f90 2023-11-24T06:21:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.989+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f93ec00cd20 con 0x7f93f8073f90 2023-11-24T06:21:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.989+0000 7f93df7fe700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f93e406aff0 0x7f93e406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.989+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f93ec089520 con 0x7f93f8073f90 2023-11-24T06:21:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.989+0000 7f93f67fc700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f93e406aff0 0x7f93e406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.989+0000 7f93df7fe700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f93e4070e50 0x7f93e4073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.989+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 --> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93e4073940 con 0x7f93e4070e50 2023-11-24T06:21:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.989+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93ec08bbe0 con 0x7f93f8073f90 2023-11-24T06:21:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.989+0000 7f93f77fe700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f93e4070e50 0x7f93e4073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.990+0000 7f93f67fc700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f93e406aff0 0x7f93e406d4b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f93f81f6bc0 tx=0x7f93e8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.990+0000 7f93f77fe700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f93e4070e50 0x7f93e4073270 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:27.990+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 <== osd.0 v2:172.21.15.89:6800/1704280696 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f93e4073940 con 0x7f93e4070e50 2023-11-24T06:21:28.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 --> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f93f80c6480 con 0x7f93e4070e50 2023-11-24T06:21:28.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93df7fe700 1 -- 172.21.15.89:0/3264632189 <== osd.0 v2:172.21.15.89:6800/1704280696 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f93f80c6480 con 0x7f93e4070e50 2023-11-24T06:21:28.008 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:21:28.009 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:28.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f93e4070e50 msgr2=0x7f93e4073270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6800/1704280696,v1:172.21.15.89:6801/1704280696] conn(0x7f93e4070e50 0x7f93e4073270 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f93e406aff0 msgr2=0x7f93e406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f93e406aff0 0x7f93e406d4b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f93f81f6bc0 tx=0x7f93e8008030 comp rx=0 tx=0).stop 2023-11-24T06:21:28.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8073f90 msgr2=0x7f93f815d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 --2- 172.21.15.89:0/3264632189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8073f90 0x7f93f815d9f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00e9b0 tx=0x7f93ec00ed70 comp rx=0 tx=0).stop 2023-11-24T06:21:28.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93f77fe700 1 -- 172.21.15.89:0/3264632189 reap_dead start 2023-11-24T06:21:28.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 shutdown_connections 2023-11-24T06:21:28.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.005+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 >> 172.21.15.89:0/3264632189 conn(0x7f93f80d0b40 msgr2=0x7f93f80af570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:28.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.006+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 shutdown_connections 2023-11-24T06:21:28.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.006+0000 7f93fd645700 1 -- 172.21.15.89:0/3264632189 wait complete. 2023-11-24T06:21:28.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-11-24T06:21:28.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.138+0000 7f34c5366700 1 Processor -- start 2023-11-24T06:21:28.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.138+0000 7f34c5366700 1 -- start start 2023-11-24T06:21:28.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34c5366700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f34c015e1e0 0x7f34c00d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34c5366700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c00d3a00 0x7f34c00d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34c5366700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f34c00d0db0 0x7f34c006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34c5366700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c00d3150 con 0x7f34c015e1e0 2023-11-24T06:21:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34c5366700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c00d1240 con 0x7f34c00d0db0 2023-11-24T06:21:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34c5366700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c015e5c0 con 0x7f34c00d3a00 2023-11-24T06:21:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f34c015e1e0 0x7f34c00d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f34c015e1e0 0x7f34c00d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39710/0 (socket says 172.21.15.89:39710) 2023-11-24T06:21:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 -- 172.21.15.89:0/3614779113 learned_addr learned my addr 172.21.15.89:0/3614779113 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34be7fc700 1 --2- 172.21.15.89:0/3614779113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c00d3a00 0x7f34c00d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34bf7fe700 1 --2- 172.21.15.89:0/3614779113 >> v2:172.21.15.123:3300/0 conn(0x7f34c00d0db0 0x7f34c006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 -- 172.21.15.89:0/3614779113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c00d3a00 msgr2=0x7f34c00d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 --2- 172.21.15.89:0/3614779113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c00d3a00 0x7f34c00d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 -- 172.21.15.89:0/3614779113 >> v2:172.21.15.123:3300/0 conn(0x7f34c00d0db0 msgr2=0x7f34c006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 --2- 172.21.15.89:0/3614779113 >> v2:172.21.15.123:3300/0 conn(0x7f34c00d0db0 0x7f34c006aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 -- 172.21.15.89:0/3614779113 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c00d20b0 con 0x7f34c015e1e0 2023-11-24T06:21:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34bf7fe700 1 --2- 172.21.15.89:0/3614779113 >> v2:172.21.15.123:3300/0 conn(0x7f34c00d0db0 0x7f34c006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.139+0000 7f34beffd700 1 --2- 172.21.15.89:0/3614779113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c015e1e0 0x7f34c00d34c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f34c00ce440 tx=0x7f34b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb9cf72c2dfea7d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34bdffb700 1 -- 172.21.15.89:0/3614779113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4011830 con 0x7f34c015e1e0 2023-11-24T06:21:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34bdffb700 1 -- 172.21.15.89:0/3614779113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b4011990 con 0x7f34c015e1e0 2023-11-24T06:21:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34bdffb700 1 -- 172.21.15.89:0/3614779113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4011d20 con 0x7f34c015e1e0 2023-11-24T06:21:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 -- 172.21.15.89:0/3614779113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c015e1e0 msgr2=0x7f34c00d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 --2- 172.21.15.89:0/3614779113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c015e1e0 0x7f34c00d34c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f34c00ce440 tx=0x7f34b4009c00 comp rx=0 tx=0).stop 2023-11-24T06:21:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 -- 172.21.15.89:0/3614779113 shutdown_connections 2023-11-24T06:21:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 --2- 172.21.15.89:0/3614779113 >> v2:172.21.15.123:3300/0 conn(0x7f34c00d0db0 0x7f34c006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 --2- 172.21.15.89:0/3614779113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c015e1e0 0x7f34c00d34c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 --2- 172.21.15.89:0/3614779113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c00d3a00 0x7f34c00d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 -- 172.21.15.89:0/3614779113 >> 172.21.15.89:0/3614779113 conn(0x7f34c00b5b50 msgr2=0x7f34c00b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 -- 172.21.15.89:0/3614779113 shutdown_connections 2023-11-24T06:21:28.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.140+0000 7f34c5366700 1 -- 172.21.15.89:0/3614779113 wait complete. 2023-11-24T06:21:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.141+0000 7f34c5366700 1 Processor -- start 2023-11-24T06:21:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34c5366700 1 -- start start 2023-11-24T06:21:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34c5366700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c00d0db0 0x7f34c00ad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34c5366700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c00d3a00 0x7f34c00ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34c5366700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c015e1e0 0x7f34c014e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34c5366700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c00aa840 con 0x7f34c00d0db0 2023-11-24T06:21:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34c5366700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c00ccba0 con 0x7f34c00d3a00 2023-11-24T06:21:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34c5366700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c014c680 con 0x7f34c015e1e0 2023-11-24T06:21:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34beffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c00d0db0 0x7f34c00ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34beffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c00d0db0 0x7f34c00ad180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39716/0 (socket says 172.21.15.89:39716) 2023-11-24T06:21:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34beffd700 1 -- 172.21.15.89:0/2165010536 learned_addr learned my addr 172.21.15.89:0/2165010536 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34be7fc700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c00d3a00 0x7f34c00ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.142+0000 7f34bf7fe700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c015e1e0 0x7f34c014e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34beffd700 1 -- 172.21.15.89:0/2165010536 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c015e1e0 msgr2=0x7f34c014e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34beffd700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c015e1e0 0x7f34c014e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34beffd700 1 -- 172.21.15.89:0/2165010536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c00d3a00 msgr2=0x7f34c00ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34beffd700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c00d3a00 0x7f34c00ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34beffd700 1 -- 172.21.15.89:0/2165010536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b4009550 con 0x7f34c00d0db0 2023-11-24T06:21:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34beffd700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c00d0db0 0x7f34c00ad180 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f34b4000c00 tx=0x7f34b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34a77fe700 1 -- 172.21.15.89:0/2165010536 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b40036a0 con 0x7f34c00d0db0 2023-11-24T06:21:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34a77fe700 1 -- 172.21.15.89:0/2165010536 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b401e070 con 0x7f34c00d0db0 2023-11-24T06:21:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c014c960 con 0x7f34c00d0db0 2023-11-24T06:21:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34a77fe700 1 -- 172.21.15.89:0/2165010536 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4003a40 con 0x7f34c00d0db0 2023-11-24T06:21:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.143+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c014ea50 con 0x7f34c00d0db0 2023-11-24T06:21:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.144+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c00cce80 con 0x7f34c00d0db0 2023-11-24T06:21:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.144+0000 7f34a77fe700 1 -- 172.21.15.89:0/2165010536 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f34b4003ba0 con 0x7f34c00d0db0 2023-11-24T06:21:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.145+0000 7f34a77fe700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f34ac06aff0 0x7f34ac06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.145+0000 7f34a77fe700 1 -- 172.21.15.89:0/2165010536 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f34b408a380 con 0x7f34c00d0db0 2023-11-24T06:21:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.145+0000 7f34be7fc700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f34ac06aff0 0x7f34ac06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.145+0000 7f34be7fc700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f34ac06aff0 0x7f34ac06d4b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f34c00d1c30 tx=0x7f34b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.149+0000 7f34a77fe700 1 -- 172.21.15.89:0/2165010536 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34b4055620 con 0x7f34c00d0db0 2023-11-24T06:21:28.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.340+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f34c006c000 con 0x7f34ac06aff0 2023-11-24T06:21:28.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.340+0000 7f34a77fe700 1 -- 172.21.15.89:0/2165010536 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f34c006c000 con 0x7f34ac06aff0 2023-11-24T06:21:28.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f34ac06aff0 msgr2=0x7f34ac06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f34ac06aff0 0x7f34ac06d4b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f34c00d1c30 tx=0x7f34b0008040 comp rx=0 tx=0).stop 2023-11-24T06:21:28.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c00d0db0 msgr2=0x7f34c00ad180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c00d0db0 0x7f34c00ad180 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f34b4000c00 tx=0x7f34b4004750 comp rx=0 tx=0).stop 2023-11-24T06:21:28.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 shutdown_connections 2023-11-24T06:21:28.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34c015e1e0 0x7f34c014e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34c00d0db0 0x7f34c00ad180 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f34ac06aff0 0x7f34ac06d4b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 --2- 172.21.15.89:0/2165010536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c00d3a00 0x7f34c00ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 >> 172.21.15.89:0/2165010536 conn(0x7f34c00b5b50 msgr2=0x7f34c00b8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:28.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 shutdown_connections 2023-11-24T06:21:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.343+0000 7f34c5366700 1 -- 172.21.15.89:0/2165010536 wait complete. 2023-11-24T06:21:28.348 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 0 to deep-scrub 2023-11-24T06:21:28.357 INFO:tasks.ceph:Scrubbing osd.1 2023-11-24T06:21:28.358 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:21:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.482+0000 7f158a334700 1 Processor -- start 2023-11-24T06:21:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.482+0000 7f158a334700 1 -- start start 2023-11-24T06:21:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f158a334700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f158415a510 0x7f15840622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f158a334700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f15840cde90 0x7f15840ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f158a334700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15840ad1e0 0x7f15840ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f158a334700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15840aebd0 con 0x7f158415a510 2023-11-24T06:21:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f158a334700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15840ce860 con 0x7f15840cde90 2023-11-24T06:21:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f158a334700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15840aeff0 con 0x7f15840ad1e0 2023-11-24T06:21:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f158415a510 0x7f15840622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f158415a510 0x7f15840622a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39724/0 (socket says 172.21.15.89:39724) 2023-11-24T06:21:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 -- 172.21.15.89:0/2513931483 learned_addr learned my addr 172.21.15.89:0/2513931483 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f15888d1700 1 --2- 172.21.15.89:0/2513931483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15840ad1e0 0x7f15840ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f15837fe700 1 --2- 172.21.15.89:0/2513931483 >> v2:172.21.15.123:3300/0 conn(0x7f15840cde90 0x7f15840ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 -- 172.21.15.89:0/2513931483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15840ad1e0 msgr2=0x7f15840ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 --2- 172.21.15.89:0/2513931483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15840ad1e0 0x7f15840ad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 -- 172.21.15.89:0/2513931483 >> v2:172.21.15.123:3300/0 conn(0x7f15840cde90 msgr2=0x7f15840ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 --2- 172.21.15.89:0/2513931483 >> v2:172.21.15.123:3300/0 conn(0x7f15840cde90 0x7f15840ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 -- 172.21.15.89:0/2513931483 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15840b0c90 con 0x7f158415a510 2023-11-24T06:21:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1583fff700 1 --2- 172.21.15.89:0/2513931483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f158415a510 0x7f15840622a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f15840cfaf0 tx=0x7f1574009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d4e8624a78b1d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1582ffd700 1 -- 172.21.15.89:0/2513931483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157400b870 con 0x7f158415a510 2023-11-24T06:21:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1582ffd700 1 -- 172.21.15.89:0/2513931483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157400b9d0 con 0x7f158415a510 2023-11-24T06:21:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.483+0000 7f1582ffd700 1 -- 172.21.15.89:0/2513931483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157400bd60 con 0x7f158415a510 2023-11-24T06:21:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 -- 172.21.15.89:0/2513931483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f158415a510 msgr2=0x7f15840622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 --2- 172.21.15.89:0/2513931483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f158415a510 0x7f15840622a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f15840cfaf0 tx=0x7f1574009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 -- 172.21.15.89:0/2513931483 shutdown_connections 2023-11-24T06:21:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 --2- 172.21.15.89:0/2513931483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15840ad1e0 0x7f15840ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 --2- 172.21.15.89:0/2513931483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f158415a510 0x7f15840622a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 --2- 172.21.15.89:0/2513931483 >> v2:172.21.15.123:3300/0 conn(0x7f15840cde90 0x7f15840ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 -- 172.21.15.89:0/2513931483 >> 172.21.15.89:0/2513931483 conn(0x7f1584072710 msgr2=0x7f1584072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 -- 172.21.15.89:0/2513931483 shutdown_connections 2023-11-24T06:21:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.484+0000 7f158a334700 1 -- 172.21.15.89:0/2513931483 wait complete. 2023-11-24T06:21:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.485+0000 7f158a334700 1 Processor -- start 2023-11-24T06:21:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.485+0000 7f158a334700 1 -- start start 2023-11-24T06:21:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f158a334700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15840ad1e0 0x7f158415a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f158a334700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15840cde90 0x7f158414af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f158a334700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f158415a510 0x7f15840c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f158a334700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158414b440 con 0x7f15840ad1e0 2023-11-24T06:21:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f158a334700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15840c4f50 con 0x7f15840cde90 2023-11-24T06:21:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f158a334700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15840c5230 con 0x7f158415a510 2023-11-24T06:21:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15840ad1e0 0x7f158415a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15840ad1e0 0x7f158415a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39728/0 (socket says 172.21.15.89:39728) 2023-11-24T06:21:28.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 -- 172.21.15.89:0/1406216890 learned_addr learned my addr 172.21.15.89:0/1406216890 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:28.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f15837fe700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15840cde90 0x7f158414af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f15888d1700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f158415a510 0x7f15840c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 -- 172.21.15.89:0/1406216890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f158415a510 msgr2=0x7f15840c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f158415a510 0x7f15840c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 -- 172.21.15.89:0/1406216890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15840cde90 msgr2=0x7f158414af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15840cde90 0x7f158414af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 -- 172.21.15.89:0/1406216890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1574009550 con 0x7f15840ad1e0 2023-11-24T06:21:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f15837fe700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15840cde90 0x7f158414af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f15888d1700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f158415a510 0x7f15840c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.486+0000 7f1583fff700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15840ad1e0 0x7f158415a8f0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1574000c00 tx=0x7f1574004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.487+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574004520 con 0x7f15840ad1e0 2023-11-24T06:21:28.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.487+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157401e070 con 0x7f15840ad1e0 2023-11-24T06:21:28.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.487+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15840c5510 con 0x7f15840ad1e0 2023-11-24T06:21:28.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.487+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15740038c0 con 0x7f15840ad1e0 2023-11-24T06:21:28.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.487+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15840c5ad0 con 0x7f15840ad1e0 2023-11-24T06:21:28.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.487+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15840ae510 con 0x7f15840ad1e0 2023-11-24T06:21:28.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.488+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f1574003a20 con 0x7f15840ad1e0 2023-11-24T06:21:28.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.488+0000 7f1580ff9700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f157006aff0 0x7f157006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.489+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1574020070 con 0x7f15840ad1e0 2023-11-24T06:21:28.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.489+0000 7f15837fe700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f157006aff0 0x7f157006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.489+0000 7f1580ff9700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f1570070ea0 0x7f15700732c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.489+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 --> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1570073990 con 0x7f1570070ea0 2023-11-24T06:21:28.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.489+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f157408b2b0 con 0x7f15840ad1e0 2023-11-24T06:21:28.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.489+0000 7f15888d1700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f1570070ea0 0x7f15700732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.489+0000 7f15837fe700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f157006aff0 0x7f157006d4b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f15841583d0 tx=0x7f1578008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.489+0000 7f15888d1700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f1570070ea0 0x7f15700732c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.490+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 <== osd.1 v2:172.21.15.89:6808/4291702656 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1570073990 con 0x7f1570070ea0 2023-11-24T06:21:28.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 --> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f15840ca460 con 0x7f1570070ea0 2023-11-24T06:21:28.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f1580ff9700 1 -- 172.21.15.89:0/1406216890 <== osd.1 v2:172.21.15.89:6808/4291702656 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f15840ca460 con 0x7f1570070ea0 2023-11-24T06:21:28.508 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:28.508 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:21:28.509 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:28.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f1570070ea0 msgr2=0x7f15700732c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6808/4291702656,v1:172.21.15.89:6809/4291702656] conn(0x7f1570070ea0 0x7f15700732c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f157006aff0 msgr2=0x7f157006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f157006aff0 0x7f157006d4b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f15841583d0 tx=0x7f1578008040 comp rx=0 tx=0).stop 2023-11-24T06:21:28.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15840ad1e0 msgr2=0x7f158415a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 --2- 172.21.15.89:0/1406216890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15840ad1e0 0x7f158415a8f0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1574000c00 tx=0x7f1574004750 comp rx=0 tx=0).stop 2023-11-24T06:21:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f15888d1700 1 -- 172.21.15.89:0/1406216890 reap_dead start 2023-11-24T06:21:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 shutdown_connections 2023-11-24T06:21:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 >> 172.21.15.89:0/1406216890 conn(0x7f1584072710 msgr2=0x7f15840af420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 shutdown_connections 2023-11-24T06:21:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.505+0000 7f158a334700 1 -- 172.21.15.89:0/1406216890 wait complete. 2023-11-24T06:21:28.520 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-11-24T06:21:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.637+0000 7fe369460700 1 Processor -- start 2023-11-24T06:21:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.637+0000 7fe369460700 1 -- start start 2023-11-24T06:21:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.638+0000 7fe369460700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe36415dfc0 0x7fe3640d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.638+0000 7fe369460700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe3640b2fc0 0x7fe3640d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.638+0000 7fe369460700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe3640b2a70 0x7fe364161b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.638+0000 7fe369460700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3640d2d10 con 0x7fe3640b2fc0 2023-11-24T06:21:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.638+0000 7fe369460700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3640cefd0 con 0x7fe3640b2a70 2023-11-24T06:21:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.638+0000 7fe369460700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3640cf2b0 con 0x7fe36415dfc0 2023-11-24T06:21:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.638+0000 7fe3627fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe3640b2fc0 0x7fe3640d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3627fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe3640b2fc0 0x7fe3640d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39738/0 (socket says 172.21.15.89:39738) 2023-11-24T06:21:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3627fc700 1 -- 172.21.15.89:0/1171778273 learned_addr learned my addr 172.21.15.89:0/1171778273 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3627fc700 1 -- 172.21.15.89:0/1171778273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe36415dfc0 msgr2=0x7fe3640d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe362ffd700 1 --2- 172.21.15.89:0/1171778273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe36415dfc0 0x7fe3640d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3637fe700 1 --2- 172.21.15.89:0/1171778273 >> v2:172.21.15.123:3300/0 conn(0x7fe3640b2a70 0x7fe364161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3627fc700 1 --2- 172.21.15.89:0/1171778273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe36415dfc0 0x7fe3640d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3627fc700 1 -- 172.21.15.89:0/1171778273 >> v2:172.21.15.123:3300/0 conn(0x7fe3640b2a70 msgr2=0x7fe364161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3627fc700 1 --2- 172.21.15.89:0/1171778273 >> v2:172.21.15.123:3300/0 conn(0x7fe3640b2a70 0x7fe364161b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3627fc700 1 -- 172.21.15.89:0/1171778273 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3640d2fb0 con 0x7fe3640b2fc0 2023-11-24T06:21:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe362ffd700 1 --2- 172.21.15.89:0/1171778273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe36415dfc0 0x7fe3640d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3637fe700 1 --2- 172.21.15.89:0/1171778273 >> v2:172.21.15.123:3300/0 conn(0x7fe3640b2a70 0x7fe364161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe3627fc700 1 --2- 172.21.15.89:0/1171778273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2fc0 0x7fe3640d13a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe3640b2a00 tx=0x7fe354009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf61f84e748035 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe361ffb700 1 -- 172.21.15.89:0/1171778273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400b870 con 0x7fe3640b2fc0 2023-11-24T06:21:28.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe361ffb700 1 -- 172.21.15.89:0/1171778273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35400b9d0 con 0x7fe3640b2fc0 2023-11-24T06:21:28.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.639+0000 7fe361ffb700 1 -- 172.21.15.89:0/1171778273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400bd60 con 0x7fe3640b2fc0 2023-11-24T06:21:28.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 -- 172.21.15.89:0/1171778273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2fc0 msgr2=0x7fe3640d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 --2- 172.21.15.89:0/1171778273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2fc0 0x7fe3640d13a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe3640b2a00 tx=0x7fe354009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 -- 172.21.15.89:0/1171778273 shutdown_connections 2023-11-24T06:21:28.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 --2- 172.21.15.89:0/1171778273 >> v2:172.21.15.123:3300/0 conn(0x7fe3640b2a70 0x7fe364161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 --2- 172.21.15.89:0/1171778273 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe36415dfc0 0x7fe3640d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 --2- 172.21.15.89:0/1171778273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2fc0 0x7fe3640d13a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 -- 172.21.15.89:0/1171778273 >> 172.21.15.89:0/1171778273 conn(0x7fe36406b410 msgr2=0x7fe36406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:28.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 -- 172.21.15.89:0/1171778273 shutdown_connections 2023-11-24T06:21:28.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.640+0000 7fe369460700 1 -- 172.21.15.89:0/1171778273 wait complete. 2023-11-24T06:21:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.641+0000 7fe369460700 1 Processor -- start 2023-11-24T06:21:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.641+0000 7fe369460700 1 -- start start 2023-11-24T06:21:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe369460700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2a70 0x7fe364158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe369460700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3640b2fc0 0x7fe364160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe369460700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe36415dfc0 0x7fe3640d4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe369460700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36415f570 con 0x7fe3640b2a70 2023-11-24T06:21:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe369460700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe364158a40 con 0x7fe36415dfc0 2023-11-24T06:21:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe369460700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36415f780 con 0x7fe3640b2fc0 2023-11-24T06:21:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2a70 0x7fe364158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2a70 0x7fe364158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39742/0 (socket says 172.21.15.89:39742) 2023-11-24T06:21:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 -- 172.21.15.89:0/2147685431 learned_addr learned my addr 172.21.15.89:0/2147685431 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe3627fc700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3640b2fc0 0x7fe364160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe3637fe700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe36415dfc0 0x7fe3640d4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 -- 172.21.15.89:0/2147685431 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3640b2fc0 msgr2=0x7fe364160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3640b2fc0 0x7fe364160f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 -- 172.21.15.89:0/2147685431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe36415dfc0 msgr2=0x7fe3640d4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe36415dfc0 0x7fe3640d4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 -- 172.21.15.89:0/2147685431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354009550 con 0x7fe3640b2a70 2023-11-24T06:21:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe362ffd700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2a70 0x7fe364158500 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c780 tx=0x7fe35800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe34b7fe700 1 -- 172.21.15.89:0/2147685431 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358011800 con 0x7fe3640b2a70 2023-11-24T06:21:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe34b7fe700 1 -- 172.21.15.89:0/2147685431 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe358011960 con 0x7fe3640b2a70 2023-11-24T06:21:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe34b7fe700 1 -- 172.21.15.89:0/2147685431 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358011cd0 con 0x7fe3640b2a70 2023-11-24T06:21:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.642+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36415fa60 con 0x7fe3640b2a70 2023-11-24T06:21:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.643+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe364160020 con 0x7fe3640b2a70 2023-11-24T06:21:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.643+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe364067f60 con 0x7fe3640b2a70 2023-11-24T06:21:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.644+0000 7fe34b7fe700 1 -- 172.21.15.89:0/2147685431 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fe358021350 con 0x7fe3640b2a70 2023-11-24T06:21:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.644+0000 7fe34b7fe700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe35006aff0 0x7fe35006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.644+0000 7fe34b7fe700 1 -- 172.21.15.89:0/2147685431 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe358089560 con 0x7fe3640b2a70 2023-11-24T06:21:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.644+0000 7fe3627fc700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe35006aff0 0x7fe35006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.645+0000 7fe3627fc700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe35006aff0 0x7fe35006d4b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe354000c00 tx=0x7fe354008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.649+0000 7fe34b7fe700 1 -- 172.21.15.89:0/2147685431 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe358054700 con 0x7fe3640b2a70 2023-11-24T06:21:28.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.838+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe36415e3a0 con 0x7fe35006aff0 2023-11-24T06:21:28.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.839+0000 7fe34b7fe700 1 -- 172.21.15.89:0/2147685431 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe35400bcc0 con 0x7fe35006aff0 2023-11-24T06:21:28.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe35006aff0 msgr2=0x7fe35006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe35006aff0 0x7fe35006d4b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe354000c00 tx=0x7fe354008030 comp rx=0 tx=0).stop 2023-11-24T06:21:28.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2a70 msgr2=0x7fe364158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2a70 0x7fe364158500 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c780 tx=0x7fe35800cb40 comp rx=0 tx=0).stop 2023-11-24T06:21:28.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 shutdown_connections 2023-11-24T06:21:28.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fe35006aff0 0x7fe35006d4b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe36415dfc0 0x7fe3640d4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3640b2a70 0x7fe364158500 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 --2- 172.21.15.89:0/2147685431 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3640b2fc0 0x7fe364160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 >> 172.21.15.89:0/2147685431 conn(0x7fe36406b410 msgr2=0x7fe3640b5280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:28.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 shutdown_connections 2023-11-24T06:21:28.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.841+0000 7fe369460700 1 -- 172.21.15.89:0/2147685431 wait complete. 2023-11-24T06:21:28.847 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 1 to deep-scrub 2023-11-24T06:21:28.856 INFO:tasks.ceph:Scrubbing osd.2 2023-11-24T06:21:28.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:21:28.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.977+0000 7f323ca79700 1 Processor -- start 2023-11-24T06:21:28.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.977+0000 7f323ca79700 1 -- start start 2023-11-24T06:21:28.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323ca79700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380b01e0 0x7f323806e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323ca79700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f32380746a0 0x7f32380b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323ca79700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32380b5e40 con 0x7f32380b59c0 2023-11-24T06:21:28.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323ca79700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32381470a0 con 0x7f32380746a0 2023-11-24T06:21:28.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323ca79700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32380b05c0 con 0x7f32380b01e0 2023-11-24T06:21:28.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f3236d9d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f32380746a0 0x7f32380b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323659c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380b01e0 0x7f323806e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f3236d9d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f32380746a0 0x7f32380b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:54864/0 (socket says 172.21.15.89:54864) 2023-11-24T06:21:28.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323659c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380b01e0 0x7f323806e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.89:48626/0 (socket says 172.21.15.89:48626) 2023-11-24T06:21:28.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f3236d9d700 1 -- 172.21.15.89:0/614463811 learned_addr learned my addr 172.21.15.89:0/614463811 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:28.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323659c700 1 -- 172.21.15.89:0/614463811 >> v2:172.21.15.123:3300/0 conn(0x7f32380746a0 msgr2=0x7f32380b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:28.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323659c700 1 --2- 172.21.15.89:0/614463811 >> v2:172.21.15.123:3300/0 conn(0x7f32380746a0 0x7f32380b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323659c700 1 -- 172.21.15.89:0/614463811 >> v1:172.21.15.123:6789/0 conn(0x7f32380b59c0 legacy=0x7f3238073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f323659c700 1 -- 172.21.15.89:0/614463811 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32380b4250 con 0x7f32380b01e0 2023-11-24T06:21:28.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.978+0000 7f3236d9d700 1 --2- 172.21.15.89:0/614463811 >> v2:172.21.15.123:3300/0 conn(0x7f32380746a0 0x7f32380b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:21:28.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323659c700 1 --2- 172.21.15.89:0/614463811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380b01e0 0x7f323806e9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3228009840 tx=0x7f3228009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc789b6832663439 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323559a700 1 -- 172.21.15.89:0/614463811 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228011830 con 0x7f32380b01e0 2023-11-24T06:21:28.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323559a700 1 -- 172.21.15.89:0/614463811 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3228011990 con 0x7f32380b01e0 2023-11-24T06:21:28.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323559a700 1 -- 172.21.15.89:0/614463811 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228011d20 con 0x7f32380b01e0 2023-11-24T06:21:28.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323ca79700 1 -- 172.21.15.89:0/614463811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380b01e0 msgr2=0x7f323806e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323ca79700 1 --2- 172.21.15.89:0/614463811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380b01e0 0x7f323806e9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3228009840 tx=0x7f3228009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:28.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323ca79700 1 -- 172.21.15.89:0/614463811 shutdown_connections 2023-11-24T06:21:28.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323ca79700 1 --2- 172.21.15.89:0/614463811 >> v2:172.21.15.123:3300/0 conn(0x7f32380746a0 0x7f32380b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323ca79700 1 --2- 172.21.15.89:0/614463811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380b01e0 0x7f323806e9c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323ca79700 1 -- 172.21.15.89:0/614463811 >> 172.21.15.89:0/614463811 conn(0x7f32380d0b40 msgr2=0x7f32380d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:28.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323ca79700 1 -- 172.21.15.89:0/614463811 shutdown_connections 2023-11-24T06:21:28.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.979+0000 7f323ca79700 1 -- 172.21.15.89:0/614463811 wait complete. 2023-11-24T06:21:28.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.981+0000 7f323ca79700 1 Processor -- start 2023-11-24T06:21:28.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.981+0000 7f323ca79700 1 -- start start 2023-11-24T06:21:28.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.981+0000 7f323ca79700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380746a0 0x7f323815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.981+0000 7f323ca79700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32380b01e0 0x7f32381314e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f323ca79700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32380b59c0 0x7f3238131a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f323ca79700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323815b560 con 0x7f32380b59c0 2023-11-24T06:21:28.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f323ca79700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323815e9f0 con 0x7f32380b01e0 2023-11-24T06:21:28.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f323ca79700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323815e430 con 0x7f32380746a0 2023-11-24T06:21:28.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32380b59c0 0x7f3238131a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32380b59c0 0x7f3238131a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39758/0 (socket says 172.21.15.89:39758) 2023-11-24T06:21:28.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 -- 172.21.15.89:0/2287013859 learned_addr learned my addr 172.21.15.89:0/2287013859 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:28.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f323659c700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380746a0 0x7f323815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3235d9b700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32380b01e0 0x7f32381314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 -- 172.21.15.89:0/2287013859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380746a0 msgr2=0x7f323815b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32380746a0 0x7f323815b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 -- 172.21.15.89:0/2287013859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32380b01e0 msgr2=0x7f32381314e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:28.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32380b01e0 0x7f32381314e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:28.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 -- 172.21.15.89:0/2287013859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3230009b30 con 0x7f32380b59c0 2023-11-24T06:21:28.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3236d9d700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32380b59c0 0x7f3238131a20 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f323000efc0 tx=0x7f323000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:28.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230018070 con 0x7f32380b59c0 2023-11-24T06:21:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323000f040 con 0x7f32380b59c0 2023-11-24T06:21:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230003c40 con 0x7f32380b59c0 2023-11-24T06:21:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32280094a0 con 0x7f32380b59c0 2023-11-24T06:21:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.982+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323815ed00 con 0x7f32380b59c0 2023-11-24T06:21:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.983+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f323806e1c0 con 0x7f32380b59c0 2023-11-24T06:21:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.984+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f323000c880 con 0x7f32380b59c0 2023-11-24T06:21:28.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.984+0000 7f3226ffd700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f321806aff0 0x7f321806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.984+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f32300895f0 con 0x7f32380b59c0 2023-11-24T06:21:28.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.984+0000 7f323659c700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f321806aff0 0x7f321806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:28.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.984+0000 7f3226ffd700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7f3218070e50 0x7f3218073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:28.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.984+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 --> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3218073940 con 0x7f3218070e50 2023-11-24T06:21:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.984+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f323008bcb0 con 0x7f32380b59c0 2023-11-24T06:21:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.984+0000 7f3235d9b700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7f3218070e50 0x7f3218073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.985+0000 7f323659c700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f321806aff0 0x7f321806d4b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3228000c00 tx=0x7f3228008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.985+0000 7f3235d9b700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7f3218070e50 0x7f3218073270 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:28.985+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 <== osd.2 v2:172.21.15.89:6816/341459167 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f3218073940 con 0x7f3218070e50 2023-11-24T06:21:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.002+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 --> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f323806dee0 con 0x7f3218070e50 2023-11-24T06:21:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.002+0000 7f3226ffd700 1 -- 172.21.15.89:0/2287013859 <== osd.2 v2:172.21.15.89:6816/341459167 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f322c001980 con 0x7f3218070e50 2023-11-24T06:21:29.005 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:29.006 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:21:29.006 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.002+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7f3218070e50 msgr2=0x7f3218073270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.002+0000 7f323ca79700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6816/341459167,v1:172.21.15.89:6817/341459167] conn(0x7f3218070e50 0x7f3218073270 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f321806aff0 msgr2=0x7f321806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f323ca79700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f321806aff0 0x7f321806d4b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3228000c00 tx=0x7f3228008030 comp rx=0 tx=0).stop 2023-11-24T06:21:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32380b59c0 msgr2=0x7f3238131a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f323ca79700 1 --2- 172.21.15.89:0/2287013859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32380b59c0 0x7f3238131a20 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f323000efc0 tx=0x7f323000c3a0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f3236d9d700 1 -- 172.21.15.89:0/2287013859 reap_dead start 2023-11-24T06:21:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 shutdown_connections 2023-11-24T06:21:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 >> 172.21.15.89:0/2287013859 conn(0x7f32380d0b40 msgr2=0x7f32380b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:29.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 shutdown_connections 2023-11-24T06:21:29.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.003+0000 7f323ca79700 1 -- 172.21.15.89:0/2287013859 wait complete. 2023-11-24T06:21:29.017 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-11-24T06:21:29.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.136+0000 7fb852b82700 1 Processor -- start 2023-11-24T06:21:29.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.136+0000 7fb852b82700 1 -- start start 2023-11-24T06:21:29.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb852b82700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb84c15b200 0x7fb84c0b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb852b82700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84c0cfd40 con 0x7fb84c0b1a70 2023-11-24T06:21:29.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb852b82700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84c0affa0 con 0x7fb84c15ae20 2023-11-24T06:21:29.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb852b82700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84c0d0020 con 0x7fb84c15b200 2023-11-24T06:21:29.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84bfff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb84c15b200 0x7fb84c0b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb85111f700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb84c0b1a70 0x7fb84c164b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43272/0 (socket says 172.21.15.89:43272) 2023-11-24T06:21:29.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb85111f700 1 -- 172.21.15.89:0/621785942 learned_addr learned my addr 172.21.15.89:0/621785942 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:29.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84bfff700 1 -- 172.21.15.89:0/621785942 >> v1:172.21.15.123:6789/0 conn(0x7fb84c15ae20 legacy=0x7fb84c0b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:29.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84bfff700 1 -- 172.21.15.89:0/621785942 >> v1:172.21.15.89:6789/0 conn(0x7fb84c0b1a70 legacy=0x7fb84c164b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84bfff700 1 -- 172.21.15.89:0/621785942 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c0b0280 con 0x7fb84c15b200 2023-11-24T06:21:29.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84b7fe700 1 -- 172.21.15.89:0/621785942 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559106957 0 0) 0x7fb84c0cfd40 con 0x7fb84c0b1a70 2023-11-24T06:21:29.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84bfff700 1 --2- 172.21.15.89:0/621785942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c0b29a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb838009840 tx=0x7fb838009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a069e71d8fe398c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84b7fe700 1 -- 172.21.15.89:0/621785942 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838004230 con 0x7fb84c15b200 2023-11-24T06:21:29.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84b7fe700 1 -- 172.21.15.89:0/621785942 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838004390 con 0x7fb84c15b200 2023-11-24T06:21:29.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.137+0000 7fb84b7fe700 1 -- 172.21.15.89:0/621785942 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380036a0 con 0x7fb84c15b200 2023-11-24T06:21:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.138+0000 7fb852b82700 1 -- 172.21.15.89:0/621785942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 msgr2=0x7fb84c0b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.138+0000 7fb852b82700 1 --2- 172.21.15.89:0/621785942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c0b29a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb838009840 tx=0x7fb838009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.138+0000 7fb852b82700 1 -- 172.21.15.89:0/621785942 shutdown_connections 2023-11-24T06:21:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.138+0000 7fb852b82700 1 --2- 172.21.15.89:0/621785942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c0b29a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.138+0000 7fb852b82700 1 -- 172.21.15.89:0/621785942 >> 172.21.15.89:0/621785942 conn(0x7fb84c0b57b0 msgr2=0x7fb84c0b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.138+0000 7fb852b82700 1 -- 172.21.15.89:0/621785942 shutdown_connections 2023-11-24T06:21:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.138+0000 7fb852b82700 1 -- 172.21.15.89:0/621785942 wait complete. 2023-11-24T06:21:29.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.139+0000 7fb852b82700 1 Processor -- start 2023-11-24T06:21:29.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.139+0000 7fb852b82700 1 -- start start 2023-11-24T06:21:29.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb852b82700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84c0b1a70 0x7fb84c1517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb852b82700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb84c15ae20 0x7fb84c155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb852b82700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c15dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb852b82700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c15d820 con 0x7fb84c15b200 2023-11-24T06:21:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb852b82700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c151d20 con 0x7fb84c0b1a70 2023-11-24T06:21:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb852b82700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c15cba0 con 0x7fb84c15ae20 2023-11-24T06:21:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85111f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c15dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85111f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c15dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39774/0 (socket says 172.21.15.89:39774) 2023-11-24T06:21:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85111f700 1 -- 172.21.15.89:0/1041956965 learned_addr learned my addr 172.21.15.89:0/1041956965 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb84bfff700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb84c15ae20 0x7fb84c155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85091e700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84c0b1a70 0x7fb84c1517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85111f700 1 -- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb84c15ae20 msgr2=0x7fb84c155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85111f700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb84c15ae20 0x7fb84c155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85111f700 1 -- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84c0b1a70 msgr2=0x7fb84c1517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85111f700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84c0b1a70 0x7fb84c1517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85111f700 1 -- 172.21.15.89:0/1041956965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8380094a0 con 0x7fb84c15b200 2023-11-24T06:21:29.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb85091e700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84c0b1a70 0x7fb84c1517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:29.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.140+0000 7fb84bfff700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb84c15ae20 0x7fb84c155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:29.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.141+0000 7fb85111f700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c15dc40 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb84400cba0 tx=0x7fb84400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.141+0000 7fb8497fa700 1 -- 172.21.15.89:0/1041956965 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84400ab80 con 0x7fb84c15b200 2023-11-24T06:21:29.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.141+0000 7fb8497fa700 1 -- 172.21.15.89:0/1041956965 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84400ace0 con 0x7fb84c15b200 2023-11-24T06:21:29.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.141+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c15ce80 con 0x7fb84c15b200 2023-11-24T06:21:29.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.141+0000 7fb8497fa700 1 -- 172.21.15.89:0/1041956965 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844007310 con 0x7fb84c15b200 2023-11-24T06:21:29.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.141+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c15d440 con 0x7fb84c15b200 2023-11-24T06:21:29.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.142+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84c15d960 con 0x7fb84c15b200 2023-11-24T06:21:29.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.142+0000 7fb8497fa700 1 -- 172.21.15.89:0/1041956965 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fb844007470 con 0x7fb84c15b200 2023-11-24T06:21:29.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.143+0000 7fb8497fa700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb83006aff0 0x7fb83006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.143+0000 7fb8497fa700 1 -- 172.21.15.89:0/1041956965 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb8440892a0 con 0x7fb84c15b200 2023-11-24T06:21:29.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.143+0000 7fb85091e700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb83006aff0 0x7fb83006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.143+0000 7fb85091e700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb83006aff0 0x7fb83006d4b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb840001750 tx=0x7fb840009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.147+0000 7fb8497fa700 1 -- 172.21.15.89:0/1041956965 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb84c15d960 con 0x7fb84c15b200 2023-11-24T06:21:29.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.338+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb84c0cf740 con 0x7fb83006aff0 2023-11-24T06:21:29.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.338+0000 7fb8497fa700 1 -- 172.21.15.89:0/1041956965 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb84c0cf740 con 0x7fb83006aff0 2023-11-24T06:21:29.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb83006aff0 msgr2=0x7fb83006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb83006aff0 0x7fb83006d4b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb840001750 tx=0x7fb840009040 comp rx=0 tx=0).stop 2023-11-24T06:21:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 msgr2=0x7fb84c15dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c15dc40 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb84400cba0 tx=0x7fb84400cf60 comp rx=0 tx=0).stop 2023-11-24T06:21:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 shutdown_connections 2023-11-24T06:21:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb84c15b200 0x7fb84c15dc40 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fb83006aff0 0x7fb83006d4b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84c0b1a70 0x7fb84c1517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 --2- 172.21.15.89:0/1041956965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb84c15ae20 0x7fb84c155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 >> 172.21.15.89:0/1041956965 conn(0x7fb84c0b57b0 msgr2=0x7fb84c072e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 shutdown_connections 2023-11-24T06:21:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.340+0000 7fb852b82700 1 -- 172.21.15.89:0/1041956965 wait complete. 2023-11-24T06:21:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 2 to deep-scrub 2023-11-24T06:21:29.355 INFO:tasks.ceph:Scrubbing osd.3 2023-11-24T06:21:29.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:21:29.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.478+0000 7f127a0f4700 1 Processor -- start 2023-11-24T06:21:29.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.478+0000 7f127a0f4700 1 -- start start 2023-11-24T06:21:29.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f127a0f4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 0x7f12740ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f127a0f4700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f12740b19c0 0x7f12740ad320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f127a0f4700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12740b0e50 con 0x7f12740aefc0 2023-11-24T06:21:29.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f127a0f4700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127415ea90 con 0x7f12740b19c0 2023-11-24T06:21:29.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f127a0f4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12740b02b0 con 0x7f12740af3a0 2023-11-24T06:21:29.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f1272ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 0x7f12740ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f1273fff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f12740b19c0 0x7f12740ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f1273fff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f12740b19c0 0x7f12740ad320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.89:54894/0 (socket says 172.21.15.89:54894) 2023-11-24T06:21:29.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.479+0000 7f1273fff700 1 -- 172.21.15.89:0/3372550947 learned_addr learned my addr 172.21.15.89:0/3372550947 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:29.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f1272ffd700 1 -- 172.21.15.89:0/3372550947 >> v2:172.21.15.123:3300/0 conn(0x7f12740b19c0 msgr2=0x7f12740ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:29.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f1272ffd700 1 --2- 172.21.15.89:0/3372550947 >> v2:172.21.15.123:3300/0 conn(0x7f12740b19c0 0x7f12740ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f1272ffd700 1 -- 172.21.15.89:0/3372550947 >> v1:172.21.15.123:6789/0 conn(0x7f12740aefc0 legacy=0x7f12740ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f1272ffd700 1 -- 172.21.15.89:0/3372550947 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12740b0590 con 0x7f12740af3a0 2023-11-24T06:21:29.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f12727fc700 1 -- 172.21.15.89:0/3372550947 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309269838 0 0) 0x7f12740b0e50 con 0x7f12740aefc0 2023-11-24T06:21:29.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f1273fff700 1 --2- 172.21.15.89:0/3372550947 >> v2:172.21.15.123:3300/0 conn(0x7f12740b19c0 0x7f12740ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:21:29.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f1272ffd700 1 --2- 172.21.15.89:0/3372550947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 0x7f12740ce960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f125c009840 tx=0x7f125c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e20dab8456c65dc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f12727fc700 1 -- 172.21.15.89:0/3372550947 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c011830 con 0x7f12740af3a0 2023-11-24T06:21:29.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f12727fc700 1 -- 172.21.15.89:0/3372550947 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c011990 con 0x7f12740af3a0 2023-11-24T06:21:29.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f12727fc700 1 -- 172.21.15.89:0/3372550947 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c011d20 con 0x7f12740af3a0 2023-11-24T06:21:29.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f127a0f4700 1 -- 172.21.15.89:0/3372550947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 msgr2=0x7f12740ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f127a0f4700 1 --2- 172.21.15.89:0/3372550947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 0x7f12740ce960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f125c009840 tx=0x7f125c009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:29.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f127a0f4700 1 -- 172.21.15.89:0/3372550947 shutdown_connections 2023-11-24T06:21:29.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f127a0f4700 1 --2- 172.21.15.89:0/3372550947 >> v2:172.21.15.123:3300/0 conn(0x7f12740b19c0 0x7f12740ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f127a0f4700 1 --2- 172.21.15.89:0/3372550947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 0x7f12740ce960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.480+0000 7f127a0f4700 1 -- 172.21.15.89:0/3372550947 >> 172.21.15.89:0/3372550947 conn(0x7f1274072710 msgr2=0x7f1274072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:29.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.481+0000 7f127a0f4700 1 -- 172.21.15.89:0/3372550947 shutdown_connections 2023-11-24T06:21:29.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.481+0000 7f127a0f4700 1 -- 172.21.15.89:0/3372550947 wait complete. 2023-11-24T06:21:29.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.482+0000 7f127a0f4700 1 Processor -- start 2023-11-24T06:21:29.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.482+0000 7f127a0f4700 1 -- start start 2023-11-24T06:21:29.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f127a0f4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12740aefc0 0x7f12741579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f127a0f4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 0x7f12741f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f127a0f4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12740b19c0 0x7f12740b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f127a0f4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740c8510 con 0x7f12740b19c0 2023-11-24T06:21:29.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f127a0f4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740c8760 con 0x7f12740aefc0 2023-11-24T06:21:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f127a0f4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12740c8a40 con 0x7f12740af3a0 2023-11-24T06:21:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12740b19c0 0x7f12740b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12740b19c0 0x7f12740b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39788/0 (socket says 172.21.15.89:39788) 2023-11-24T06:21:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 -- 172.21.15.89:0/844965015 learned_addr learned my addr 172.21.15.89:0/844965015 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1272ffd700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 0x7f12741f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f12737fe700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12740aefc0 0x7f12741579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 -- 172.21.15.89:0/844965015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 msgr2=0x7f12741f46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12740af3a0 0x7f12741f46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 -- 172.21.15.89:0/844965015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12740aefc0 msgr2=0x7f12741579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12740aefc0 0x7f12741579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 -- 172.21.15.89:0/844965015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c009b30 con 0x7f12740b19c0 2023-11-24T06:21:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f1273fff700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12740b19c0 0x7f12740b4f10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f126c00efc0 tx=0x7f126c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c018070 con 0x7f12740b19c0 2023-11-24T06:21:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126c00f040 con 0x7f12740b19c0 2023-11-24T06:21:29.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125c0094a0 con 0x7f12740b19c0 2023-11-24T06:21:29.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c003c40 con 0x7f12740b19c0 2023-11-24T06:21:29.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.483+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12740c9000 con 0x7f12740b19c0 2023-11-24T06:21:29.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.484+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f12740c88d0 con 0x7f12740b19c0 2023-11-24T06:21:29.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.485+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f126c00c9f0 con 0x7f12740b19c0 2023-11-24T06:21:29.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.485+0000 7f126bfff700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f125806aff0 0x7f125806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.485+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f126c089800 con 0x7f12740b19c0 2023-11-24T06:21:29.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.485+0000 7f12737fe700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f125806aff0 0x7f125806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.485+0000 7f126bfff700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f1258070e50 0x7f1258073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.485+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 --> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1258073940 con 0x7f1258070e50 2023-11-24T06:21:29.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.485+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f126c08e2c0 con 0x7f12740b19c0 2023-11-24T06:21:29.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.485+0000 7f1272ffd700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f1258070e50 0x7f1258073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.486+0000 7f12737fe700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f125806aff0 0x7f125806d4b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1264000f60 tx=0x7f1264009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.486+0000 7f1272ffd700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f1258070e50 0x7f1258073270 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.486+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 <== osd.3 v2:172.21.15.89:6824/1593118452 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1258073940 con 0x7f1258070e50 2023-11-24T06:21:29.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.501+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 --> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f12740b10f0 con 0x7f1258070e50 2023-11-24T06:21:29.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f126bfff700 1 -- 172.21.15.89:0/844965015 <== osd.3 v2:172.21.15.89:6824/1593118452 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f125c021040 con 0x7f1258070e50 2023-11-24T06:21:29.505 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:29.505 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:21:29.505 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:29.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f1258070e50 msgr2=0x7f1258073270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6824/1593118452,v1:172.21.15.89:6825/1593118452] conn(0x7f1258070e50 0x7f1258073270 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f125806aff0 msgr2=0x7f125806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f125806aff0 0x7f125806d4b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1264000f60 tx=0x7f1264009040 comp rx=0 tx=0).stop 2023-11-24T06:21:29.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12740b19c0 msgr2=0x7f12740b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 --2- 172.21.15.89:0/844965015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12740b19c0 0x7f12740b4f10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f126c00efc0 tx=0x7f126c00c3a0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f1273fff700 1 -- 172.21.15.89:0/844965015 reap_dead start 2023-11-24T06:21:29.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 shutdown_connections 2023-11-24T06:21:29.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 >> 172.21.15.89:0/844965015 conn(0x7f1274072710 msgr2=0x7f127415dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:29.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 shutdown_connections 2023-11-24T06:21:29.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.502+0000 7f127a0f4700 1 -- 172.21.15.89:0/844965015 wait complete. 2023-11-24T06:21:29.517 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-11-24T06:21:29.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.635+0000 7f51915bc700 1 Processor -- start 2023-11-24T06:21:29.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.635+0000 7f51915bc700 1 -- start start 2023-11-24T06:21:29.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f51915bc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c15e0f0 0x7f518c0b9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f51915bc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f518c0b2b60 0x7f518c0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f51915bc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f518c0cf0d0 0x7f518c161bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f51915bc700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f518c0cf530 con 0x7f518c0b2b60 2023-11-24T06:21:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f51915bc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f518c06bc00 con 0x7f518c0cf0d0 2023-11-24T06:21:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f51915bc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518c15e4d0 con 0x7f518c15e0f0 2023-11-24T06:21:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f518c0b2b60 0x7f518c0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f518c0b2b60 0x7f518c0ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39796/0 (socket says 172.21.15.89:39796) 2023-11-24T06:21:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 -- 172.21.15.89:0/1121230255 learned_addr learned my addr 172.21.15.89:0/1121230255 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:29.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 -- 172.21.15.89:0/1121230255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c15e0f0 msgr2=0x7f518c0b9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518affd700 1 --2- 172.21.15.89:0/1121230255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c15e0f0 0x7f518c0b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 --2- 172.21.15.89:0/1121230255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c15e0f0 0x7f518c0b9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518b7fe700 1 --2- 172.21.15.89:0/1121230255 >> v2:172.21.15.123:3300/0 conn(0x7f518c0cf0d0 0x7f518c161bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 -- 172.21.15.89:0/1121230255 >> v2:172.21.15.123:3300/0 conn(0x7f518c0cf0d0 msgr2=0x7f518c161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 --2- 172.21.15.89:0/1121230255 >> v2:172.21.15.123:3300/0 conn(0x7f518c0cf0d0 0x7f518c161bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 -- 172.21.15.89:0/1121230255 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c06aff0 con 0x7f518c0b2b60 2023-11-24T06:21:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.636+0000 7f518a7fc700 1 --2- 172.21.15.89:0/1121230255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0b2b60 0x7f518c0ceb90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f518c08d290 tx=0x7f517c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1487a9e75b28ea21 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f5189ffb700 1 -- 172.21.15.89:0/1121230255 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c00b870 con 0x7f518c0b2b60 2023-11-24T06:21:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f5189ffb700 1 -- 172.21.15.89:0/1121230255 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c00b9d0 con 0x7f518c0b2b60 2023-11-24T06:21:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f5189ffb700 1 -- 172.21.15.89:0/1121230255 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c00bd60 con 0x7f518c0b2b60 2023-11-24T06:21:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 -- 172.21.15.89:0/1121230255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0b2b60 msgr2=0x7f518c0ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 --2- 172.21.15.89:0/1121230255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0b2b60 0x7f518c0ceb90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f518c08d290 tx=0x7f517c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 -- 172.21.15.89:0/1121230255 shutdown_connections 2023-11-24T06:21:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 --2- 172.21.15.89:0/1121230255 >> v2:172.21.15.123:3300/0 conn(0x7f518c0cf0d0 0x7f518c161bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 --2- 172.21.15.89:0/1121230255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c15e0f0 0x7f518c0b9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 --2- 172.21.15.89:0/1121230255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0b2b60 0x7f518c0ceb90 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 -- 172.21.15.89:0/1121230255 >> 172.21.15.89:0/1121230255 conn(0x7f518c0b5ce0 msgr2=0x7f518c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 -- 172.21.15.89:0/1121230255 shutdown_connections 2023-11-24T06:21:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.637+0000 7f51915bc700 1 -- 172.21.15.89:0/1121230255 wait complete. 2023-11-24T06:21:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.638+0000 7f51915bc700 1 Processor -- start 2023-11-24T06:21:29.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f51915bc700 1 -- start start 2023-11-24T06:21:29.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f51915bc700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c0b2b60 0x7f518c0cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f51915bc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0cf0d0 0x7f518c1f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f51915bc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c15e0f0 0x7f518c160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f51915bc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c1f7230 con 0x7f518c0cf0d0 2023-11-24T06:21:29.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f51915bc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c0cc570 con 0x7f518c15e0f0 2023-11-24T06:21:29.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f51915bc700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518c0cc850 con 0x7f518c0b2b60 2023-11-24T06:21:29.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f518a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0cf0d0 0x7f518c1f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f518a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0cf0d0 0x7f518c1f6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39802/0 (socket says 172.21.15.89:39802) 2023-11-24T06:21:29.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f518a7fc700 1 -- 172.21.15.89:0/3377741865 learned_addr learned my addr 172.21.15.89:0/3377741865 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:29.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f518b7fe700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c15e0f0 0x7f518c160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.639+0000 7f518affd700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c0b2b60 0x7f518c0cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f518a7fc700 1 -- 172.21.15.89:0/3377741865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c0b2b60 msgr2=0x7f518c0cbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f518a7fc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c0b2b60 0x7f518c0cbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f518a7fc700 1 -- 172.21.15.89:0/3377741865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c15e0f0 msgr2=0x7f518c160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f518a7fc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c15e0f0 0x7f518c160f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f518a7fc700 1 -- 172.21.15.89:0/3377741865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c009550 con 0x7f518c0cf0d0 2023-11-24T06:21:29.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f518a7fc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0cf0d0 0x7f518c1f6cf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f517c014040 tx=0x7f517c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f51737fe700 1 -- 172.21.15.89:0/3377741865 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c004250 con 0x7f518c0cf0d0 2023-11-24T06:21:29.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f51737fe700 1 -- 172.21.15.89:0/3377741865 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c01e070 con 0x7f518c0cf0d0 2023-11-24T06:21:29.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c0ccb30 con 0x7f518c0cf0d0 2023-11-24T06:21:29.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f51737fe700 1 -- 172.21.15.89:0/3377741865 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c0036a0 con 0x7f518c0cf0d0 2023-11-24T06:21:29.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.640+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c0cd0f0 con 0x7f518c0cf0d0 2023-11-24T06:21:29.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.641+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c0cd280 con 0x7f518c0cf0d0 2023-11-24T06:21:29.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.642+0000 7f51737fe700 1 -- 172.21.15.89:0/3377741865 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f517c020030 con 0x7f518c0cf0d0 2023-11-24T06:21:29.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.642+0000 7f51737fe700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f517806aff0 0x7f517806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.642+0000 7f51737fe700 1 -- 172.21.15.89:0/3377741865 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f517c08a830 con 0x7f518c0cf0d0 2023-11-24T06:21:29.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.642+0000 7f518affd700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f517806aff0 0x7f517806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.643+0000 7f518affd700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f517806aff0 0x7f517806d4b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f518000ab60 tx=0x7f5180008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.646+0000 7f51737fe700 1 -- 172.21.15.89:0/3377741865 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f517c055d60 con 0x7f518c0cf0d0 2023-11-24T06:21:29.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.840+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f518c162910 con 0x7f517806aff0 2023-11-24T06:21:29.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.840+0000 7f51737fe700 1 -- 172.21.15.89:0/3377741865 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f518c162910 con 0x7f517806aff0 2023-11-24T06:21:29.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f517806aff0 msgr2=0x7f517806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f517806aff0 0x7f517806d4b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f518000ab60 tx=0x7f5180008040 comp rx=0 tx=0).stop 2023-11-24T06:21:29.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0cf0d0 msgr2=0x7f518c1f6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0cf0d0 0x7f518c1f6cf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f517c014040 tx=0x7f517c004750 comp rx=0 tx=0).stop 2023-11-24T06:21:29.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 shutdown_connections 2023-11-24T06:21:29.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f517806aff0 0x7f517806d4b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c15e0f0 0x7f518c160f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f518c0b2b60 0x7f518c0cbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 --2- 172.21.15.89:0/3377741865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f518c0cf0d0 0x7f518c1f6cf0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 >> 172.21.15.89:0/3377741865 conn(0x7f518c0b5ce0 msgr2=0x7f518c0b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:29.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 shutdown_connections 2023-11-24T06:21:29.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.842+0000 7f51915bc700 1 -- 172.21.15.89:0/3377741865 wait complete. 2023-11-24T06:21:29.851 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 3 to deep-scrub 2023-11-24T06:21:29.857 INFO:tasks.ceph:Scrubbing osd.4 2023-11-24T06:21:29.857 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:21:29.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.979+0000 7f4cb316b700 1 Processor -- start 2023-11-24T06:21:29.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.979+0000 7f4cb316b700 1 -- start start 2023-11-24T06:21:29.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cb316b700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4cac0746a0 0x7f4cac06e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cb316b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac0b59c0 0x7f4cac0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cb316b700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac0b5e50 con 0x7f4cac0b01e0 2023-11-24T06:21:29.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cb316b700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cac1470a0 con 0x7f4cac0746a0 2023-11-24T06:21:29.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cb316b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0b05c0 con 0x7f4cac0b59c0 2023-11-24T06:21:29.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cb0f07700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4cac0b01e0 0x7f4cac073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43312/0 (socket says 172.21.15.89:43312) 2023-11-24T06:21:29.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cb0f07700 1 -- 172.21.15.89:0/3922978854 learned_addr learned my addr 172.21.15.89:0/3922978854 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:29.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719875020 0 0) 0x7f4ca0001110 con 0x7f4cac0b01e0 2023-11-24T06:21:29.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c9c0035b0 con 0x7f4cac0b01e0 2023-11-24T06:21:29.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cabfff700 1 --2- 172.21.15.89:0/3922978854 >> v2:172.21.15.123:3300/0 conn(0x7f4cac0746a0 0x7f4cac06e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cb1708700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac0b59c0 0x7f4cac0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897818193 0 0) 0x7f4c9c0035b0 con 0x7f4cac0b01e0 2023-11-24T06:21:29.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac0b59c0 msgr2=0x7f4cac0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:29.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac0b59c0 0x7f4cac0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 >> v2:172.21.15.123:3300/0 conn(0x7f4cac0746a0 msgr2=0x7f4cac06e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:29.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 --2- 172.21.15.89:0/3922978854 >> v2:172.21.15.123:3300/0 conn(0x7f4cac0746a0 0x7f4cac06e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0b4250 con 0x7f4cac0b01e0 2023-11-24T06:21:29.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f4ca0001800 con 0x7f4cac0b01e0 2023-11-24T06:21:29.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.980+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 >> v1:172.21.15.89:6789/0 conn(0x7f4cac0b01e0 legacy=0x7f4cac073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c003a00 0x7f4c9c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4c9c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac0b59c0 0x7f4c9c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac1470a0 con 0x7f4c9c004850 2023-11-24T06:21:29.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0b05c0 con 0x7f4c9c003a00 2023-11-24T06:21:29.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c00ba60 con 0x7f4cac0b59c0 2023-11-24T06:21:29.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cabfff700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4c9c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cb0f07700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c003a00 0x7f4c9c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cb1708700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac0b59c0 0x7f4c9c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cabfff700 1 -- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac0b59c0 msgr2=0x7f4c9c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:29.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cabfff700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac0b59c0 0x7f4c9c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cabfff700 1 -- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c003a00 msgr2=0x7f4c9c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:29.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cabfff700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c003a00 0x7f4c9c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:29.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cabfff700 1 -- 172.21.15.89:0/3922978854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c0035b0 con 0x7f4c9c004850 2023-11-24T06:21:29.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cabfff700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4c9c004cd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009b60 tx=0x7f4c9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d32e388db5fe95 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:29.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9800e6c0 con 0x7f4c9c004850 2023-11-24T06:21:29.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9800e820 con 0x7f4c9c004850 2023-11-24T06:21:29.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.981+0000 7f4cab7fe700 1 -- 172.21.15.89:0/3922978854 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9800eb90 con 0x7f4c9c004850 2023-11-24T06:21:29.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.982+0000 7f4cb316b700 1 -- 172.21.15.89:0/3922978854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 msgr2=0x7f4c9c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:29.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.982+0000 7f4cb316b700 1 --2- 172.21.15.89:0/3922978854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4c9c004cd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009b60 tx=0x7f4c9c005880 comp rx=0 tx=0).stop 2023-11-24T06:21:29.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.982+0000 7f4cb1708700 1 -- 172.21.15.89:0/3922978854 reap_dead start 2023-11-24T06:21:29.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.982+0000 7f4cb316b700 1 -- 172.21.15.89:0/3922978854 shutdown_connections 2023-11-24T06:21:29.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.982+0000 7f4cb316b700 1 -- 172.21.15.89:0/3922978854 >> 172.21.15.89:0/3922978854 conn(0x7f4cac0d0b40 msgr2=0x7f4cac0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:29.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.982+0000 7f4cb316b700 1 -- 172.21.15.89:0/3922978854 shutdown_connections 2023-11-24T06:21:29.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.982+0000 7f4cb316b700 1 -- 172.21.15.89:0/3922978854 wait complete. 2023-11-24T06:21:29.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.984+0000 7f4cb316b700 1 Processor -- start 2023-11-24T06:21:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.984+0000 7f4cb316b700 1 -- start start 2023-11-24T06:21:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb316b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4cac1f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb316b700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac13a3d0 0x7f4cac0cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb316b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cac0cf9a0 0x7f4cac145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb316b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0cac40 con 0x7f4c9c004850 2023-11-24T06:21:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb316b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0caf20 con 0x7f4cac0cf9a0 2023-11-24T06:21:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb316b700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0cb200 con 0x7f4cac13a3d0 2023-11-24T06:21:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4cac1f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4cac1f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39820/0 (socket says 172.21.15.89:39820) 2023-11-24T06:21:29.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 -- 172.21.15.89:0/1399590172 learned_addr learned my addr 172.21.15.89:0/1399590172 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cabfff700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac13a3d0 0x7f4cac0cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb1708700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cac0cf9a0 0x7f4cac145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 -- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac13a3d0 msgr2=0x7f4cac0cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4cac13a3d0 0x7f4cac0cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 -- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cac0cf9a0 msgr2=0x7f4cac145a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cac0cf9a0 0x7f4cac145a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 -- 172.21.15.89:0/1399590172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0cb4e0 con 0x7f4c9c004850 2023-11-24T06:21:30.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.985+0000 7f4cb0f07700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4cac1f3d40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0000fb0 tx=0x7f4ca00073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.986+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0011580 con 0x7f4c9c004850 2023-11-24T06:21:30.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.986+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca0011700 con 0x7f4c9c004850 2023-11-24T06:21:30.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.986+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac0cb7c0 con 0x7f4c9c004850 2023-11-24T06:21:30.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.986+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0011ab0 con 0x7f4c9c004850 2023-11-24T06:21:30.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.986+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac13a850 con 0x7f4c9c004850 2023-11-24T06:21:30.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.986+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4cac06e1c0 con 0x7f4c9c004850 2023-11-24T06:21:30.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.987+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f4ca0011c10 con 0x7f4c9c004850 2023-11-24T06:21:30.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.987+0000 7f4ca97fa700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4c9006aff0 0x7f4c9006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.987+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4ca0089ca0 con 0x7f4c9c004850 2023-11-24T06:21:30.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.987+0000 7f4cabfff700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4c9006aff0 0x7f4c9006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.987+0000 7f4ca97fa700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7f4c90070e50 0x7f4c90073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.988+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 --> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c90073940 con 0x7f4c90070e50 2023-11-24T06:21:30.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.988+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ca008a0f0 con 0x7f4c9c004850 2023-11-24T06:21:30.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.988+0000 7f4cb1708700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7f4c90070e50 0x7f4c90073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.988+0000 7f4cabfff700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4c9006aff0 0x7f4c9006d4b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4c980090b0 tx=0x7f4c9801e040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.988+0000 7f4cb1708700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7f4c90070e50 0x7f4c90073270 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:29.989+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 <== osd.4 v2:172.21.15.123:6816/1261947525 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4c90073940 con 0x7f4c90070e50 2023-11-24T06:21:30.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.004+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 --> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4cac06dee0 con 0x7f4c90070e50 2023-11-24T06:21:30.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4ca97fa700 1 -- 172.21.15.89:0/1399590172 <== osd.4 v2:172.21.15.123:6816/1261947525 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4ca400f040 con 0x7f4c90070e50 2023-11-24T06:21:30.008 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:30.008 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:21:30.009 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:30.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7f4c90070e50 msgr2=0x7f4c90073270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.123:6816/1261947525,v1:172.21.15.123:6817/1261947525] conn(0x7f4c90070e50 0x7f4c90073270 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4c9006aff0 msgr2=0x7f4c9006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f4c9006aff0 0x7f4c9006d4b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4c980090b0 tx=0x7f4c9801e040 comp rx=0 tx=0).stop 2023-11-24T06:21:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 msgr2=0x7f4cac1f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 --2- 172.21.15.89:0/1399590172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c9c004850 0x7f4cac1f3d40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0000fb0 tx=0x7f4ca00073f0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb1708700 1 -- 172.21.15.89:0/1399590172 reap_dead start 2023-11-24T06:21:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 shutdown_connections 2023-11-24T06:21:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 >> 172.21.15.89:0/1399590172 conn(0x7f4cac0d0b40 msgr2=0x7f4cac06c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 shutdown_connections 2023-11-24T06:21:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.005+0000 7f4cb316b700 1 -- 172.21.15.89:0/1399590172 wait complete. 2023-11-24T06:21:30.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-11-24T06:21:30.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.138+0000 7f349086c700 1 Processor -- start 2023-11-24T06:21:30.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.138+0000 7f349086c700 1 -- start start 2023-11-24T06:21:30.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f349086c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c15e080 0x7f348c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f349086c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f348c06c710 0x7f348c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f349086c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348c0ae430 con 0x7f348c0af4d0 2023-11-24T06:21:30.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f349086c700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348c0af0b0 con 0x7f348c06c710 2023-11-24T06:21:30.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f349086c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0b05a0 con 0x7f348c15e080 2023-11-24T06:21:30.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348ad9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f348c06c710 0x7f348c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348ad9d700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f348c06c710 0x7f348c0afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39834/0 (socket says 172.21.15.89:39834) 2023-11-24T06:21:30.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348ad9d700 1 -- 172.21.15.89:0/1343289173 learned_addr learned my addr 172.21.15.89:0/1343289173 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348a59c700 1 --2- 172.21.15.89:0/1343289173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c15e080 0x7f348c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348ad9d700 1 -- 172.21.15.89:0/1343289173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c15e080 msgr2=0x7f348c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348ad9d700 1 --2- 172.21.15.89:0/1343289173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c15e080 0x7f348c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348ad9d700 1 -- 172.21.15.89:0/1343289173 >> v1:172.21.15.123:6789/0 conn(0x7f348c0af4d0 legacy=0x7f348c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348ad9d700 1 -- 172.21.15.89:0/1343289173 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c0ae710 con 0x7f348c06c710 2023-11-24T06:21:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.139+0000 7f348a59c700 1 --2- 172.21.15.89:0/1343289173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c15e080 0x7f348c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f348ad9d700 1 --2- 172.21.15.89:0/1343289173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c06c710 0x7f348c0afb90 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3484009e80 tx=0x7f348400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f0c0b2c8345c83 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f348959a700 1 -- 172.21.15.89:0/1343289173 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348400e070 con 0x7f348c06c710 2023-11-24T06:21:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f348959a700 1 -- 172.21.15.89:0/1343289173 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3484003a20 con 0x7f348c06c710 2023-11-24T06:21:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f348959a700 1 -- 172.21.15.89:0/1343289173 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484003dd0 con 0x7f348c06c710 2023-11-24T06:21:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f349086c700 1 -- 172.21.15.89:0/1343289173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c06c710 msgr2=0x7f348c0afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f349086c700 1 --2- 172.21.15.89:0/1343289173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c06c710 0x7f348c0afb90 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3484009e80 tx=0x7f348400b330 comp rx=0 tx=0).stop 2023-11-24T06:21:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f349086c700 1 -- 172.21.15.89:0/1343289173 shutdown_connections 2023-11-24T06:21:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f349086c700 1 --2- 172.21.15.89:0/1343289173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c06c710 0x7f348c0afb90 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f349086c700 1 --2- 172.21.15.89:0/1343289173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c15e080 0x7f348c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f349086c700 1 -- 172.21.15.89:0/1343289173 >> 172.21.15.89:0/1343289173 conn(0x7f348c071b90 msgr2=0x7f348c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:30.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f349086c700 1 -- 172.21.15.89:0/1343289173 shutdown_connections 2023-11-24T06:21:30.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.140+0000 7f349086c700 1 -- 172.21.15.89:0/1343289173 wait complete. 2023-11-24T06:21:30.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.142+0000 7f349086c700 1 Processor -- start 2023-11-24T06:21:30.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.142+0000 7f349086c700 1 -- start start 2023-11-24T06:21:30.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.142+0000 7f349086c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f348c06c710 0x7f348c153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.142+0000 7f349086c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c0af4d0 0x7f348c156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.142+0000 7f349086c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c15e080 0x7f348c1e72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.142+0000 7f349086c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c153b80 con 0x7f348c15e080 2023-11-24T06:21:30.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.142+0000 7f349086c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c157c80 con 0x7f348c06c710 2023-11-24T06:21:30.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.142+0000 7f349086c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c157f60 con 0x7f348c0af4d0 2023-11-24T06:21:30.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c15e080 0x7f348c1e72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c15e080 0x7f348c1e72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39840/0 (socket says 172.21.15.89:39840) 2023-11-24T06:21:30.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 -- 172.21.15.89:0/3934475221 learned_addr learned my addr 172.21.15.89:0/3934475221 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:30.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348a59c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f348c06c710 0x7f348c153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f3489d9b700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c0af4d0 0x7f348c156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 -- 172.21.15.89:0/3934475221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c0af4d0 msgr2=0x7f348c156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c0af4d0 0x7f348c156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 -- 172.21.15.89:0/3934475221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f348c06c710 msgr2=0x7f348c153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f348c06c710 0x7f348c153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 -- 172.21.15.89:0/3934475221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3484009b30 con 0x7f348c15e080 2023-11-24T06:21:30.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f348ad9d700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c15e080 0x7f348c1e72e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3484015040 tx=0x7f3484004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f347affd700 1 -- 172.21.15.89:0/3934475221 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484004440 con 0x7f348c15e080 2023-11-24T06:21:30.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f347affd700 1 -- 172.21.15.89:0/3934475221 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34840045c0 con 0x7f348c15e080 2023-11-24T06:21:30.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348c158240 con 0x7f348c15e080 2023-11-24T06:21:30.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f347affd700 1 -- 172.21.15.89:0/3934475221 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484004970 con 0x7f348c15e080 2023-11-24T06:21:30.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.143+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348c158800 con 0x7f348c15e080 2023-11-24T06:21:30.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.144+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348c1589c0 con 0x7f348c15e080 2023-11-24T06:21:30.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.145+0000 7f347affd700 1 -- 172.21.15.89:0/3934475221 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3484012430 con 0x7f348c15e080 2023-11-24T06:21:30.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.145+0000 7f347affd700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f346c06aff0 0x7f346c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.145+0000 7f347affd700 1 -- 172.21.15.89:0/3934475221 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f348408a8f0 con 0x7f348c15e080 2023-11-24T06:21:30.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.145+0000 7f348a59c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f346c06aff0 0x7f346c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.145+0000 7f348a59c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f346c06aff0 0x7f346c06d4b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f348c155d90 tx=0x7f3480008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.149+0000 7f347affd700 1 -- 172.21.15.89:0/3934475221 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34840595a0 con 0x7f348c15e080 2023-11-24T06:21:30.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.341+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f348c0aecd0 con 0x7f346c06aff0 2023-11-24T06:21:30.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.341+0000 7f347affd700 1 -- 172.21.15.89:0/3934475221 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f348000c040 con 0x7f346c06aff0 2023-11-24T06:21:30.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f346c06aff0 msgr2=0x7f346c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f346c06aff0 0x7f346c06d4b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f348c155d90 tx=0x7f3480008040 comp rx=0 tx=0).stop 2023-11-24T06:21:30.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c15e080 msgr2=0x7f348c1e72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c15e080 0x7f348c1e72e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3484015040 tx=0x7f3484004ad0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 shutdown_connections 2023-11-24T06:21:30.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c15e080 0x7f348c1e72e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f346c06aff0 0x7f346c06d4b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f348c06c710 0x7f348c153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 --2- 172.21.15.89:0/3934475221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f348c0af4d0 0x7f348c156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 >> 172.21.15.89:0/3934475221 conn(0x7f348c071b90 msgr2=0x7f348c0b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:30.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 shutdown_connections 2023-11-24T06:21:30.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.343+0000 7f349086c700 1 -- 172.21.15.89:0/3934475221 wait complete. 2023-11-24T06:21:30.348 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 4 to deep-scrub 2023-11-24T06:21:30.358 INFO:tasks.ceph:Scrubbing osd.5 2023-11-24T06:21:30.358 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:21:30.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.478+0000 7f64cd19c700 1 Processor -- start 2023-11-24T06:21:30.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.478+0000 7f64cd19c700 1 -- start start 2023-11-24T06:21:30.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64cd19c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f64c815e020 0x7f64c80b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64cd19c700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c80b1b80 con 0x7f64c80b1720 2023-11-24T06:21:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64cd19c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c8161c80 con 0x7f64c80cfae0 2023-11-24T06:21:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64cd19c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c8161f40 con 0x7f64c815e020 2023-11-24T06:21:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64c659c700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f64c80b1720 0x7f64c80cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43350/0 (socket says 172.21.15.89:43350) 2023-11-24T06:21:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64c659c700 1 -- 172.21.15.89:0/1845686945 learned_addr learned my addr 172.21.15.89:0/1845686945 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:30.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64c6d9d700 1 --2- 172.21.15.89:0/1845686945 >> v2:172.21.15.123:3300/0 conn(0x7f64c815e020 0x7f64c80b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446187139 0 0) 0x7f64c80b1b80 con 0x7f64c80b1720 2023-11-24T06:21:30.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.479+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b00035b0 con 0x7f64c80b1720 2023-11-24T06:21:30.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4252763918 0 0) 0x7f64b00035b0 con 0x7f64c80b1720 2023-11-24T06:21:30.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 >> v2:172.21.15.123:3300/0 conn(0x7f64c815e020 msgr2=0x7f64c80b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:30.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 --2- 172.21.15.89:0/1845686945 >> v2:172.21.15.123:3300/0 conn(0x7f64c815e020 0x7f64c80b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 >> v1:172.21.15.123:6789/0 conn(0x7f64c80cfae0 legacy=0x7f64c80af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c8162220 con 0x7f64c80b1720 2023-11-24T06:21:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1749945864 0 0) 0x7f64b8001670 con 0x7f64c80b1720 2023-11-24T06:21:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 >> v1:172.21.15.89:6789/0 conn(0x7f64c80b1720 legacy=0x7f64c80cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64b0003a00 0x7f64b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64b00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64b0005170 0x7f64b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8161f40 con 0x7f64b0004400 2023-11-24T06:21:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b000c0b0 con 0x7f64b0003a00 2023-11-24T06:21:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b000c270 con 0x7f64b0005170 2023-11-24T06:21:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c659c700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64b00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c6d9d700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64b0003a00 0x7f64b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c759e700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64b0005170 0x7f64b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c659c700 1 -- 172.21.15.89:0/1845686945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64b0005170 msgr2=0x7f64b000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c659c700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64b0005170 0x7f64b000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c659c700 1 -- 172.21.15.89:0/1845686945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64b0003a00 msgr2=0x7f64b0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c659c700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64b0003a00 0x7f64b0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c659c700 1 -- 172.21.15.89:0/1845686945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b00035b0 con 0x7f64b0004400 2023-11-24T06:21:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c759e700 1 -- 172.21.15.89:0/1845686945 reap_dead start 2023-11-24T06:21:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c659c700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64b00048d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f64b800c2f0 tx=0x7f64b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a5111b5178baac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.480+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b800ce30 con 0x7f64b0004400 2023-11-24T06:21:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b8005960 con 0x7f64b0004400 2023-11-24T06:21:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64c5d9b700 1 -- 172.21.15.89:0/1845686945 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8005d10 con 0x7f64b0004400 2023-11-24T06:21:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64cd19c700 1 -- 172.21.15.89:0/1845686945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 msgr2=0x7f64b00048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64cd19c700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64b00048d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f64b800c2f0 tx=0x7f64b0005880 comp rx=0 tx=0).stop 2023-11-24T06:21:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64cd19c700 1 -- 172.21.15.89:0/1845686945 shutdown_connections 2023-11-24T06:21:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64cd19c700 1 --2- 172.21.15.89:0/1845686945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64b00048d0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64cd19c700 1 -- 172.21.15.89:0/1845686945 >> 172.21.15.89:0/1845686945 conn(0x7f64c8072710 msgr2=0x7f64c8072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64cd19c700 1 -- 172.21.15.89:0/1845686945 shutdown_connections 2023-11-24T06:21:30.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.481+0000 7f64cd19c700 1 -- 172.21.15.89:0/1845686945 wait complete. 2023-11-24T06:21:30.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64cd19c700 1 Processor -- start 2023-11-24T06:21:30.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64cd19c700 1 -- start start 2023-11-24T06:21:30.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64cd19c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64c81535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64cd19c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64c81f6980 0x7f64c8153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64cd19c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64c8154250 0x7f64c815e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64cd19c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8161150 con 0x7f64b0004400 2023-11-24T06:21:30.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64cd19c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c81546d0 con 0x7f64c81f6980 2023-11-24T06:21:30.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64cd19c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8153b30 con 0x7f64c8154250 2023-11-24T06:21:30.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64c6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64c81535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64c6d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64c81535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39856/0 (socket says 172.21.15.89:39856) 2023-11-24T06:21:30.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64c6d9d700 1 -- 172.21.15.89:0/508268421 learned_addr learned my addr 172.21.15.89:0/508268421 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:30.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64c759e700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64c8154250 0x7f64c815e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.483+0000 7f64c659c700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64c81f6980 0x7f64c8153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64c6d9d700 1 -- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64c8154250 msgr2=0x7f64c815e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64c6d9d700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64c8154250 0x7f64c815e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64c6d9d700 1 -- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64c81f6980 msgr2=0x7f64c8153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64c6d9d700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64c81f6980 0x7f64c8153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64c6d9d700 1 -- 172.21.15.89:0/508268421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c81604d0 con 0x7f64b0004400 2023-11-24T06:21:30.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64c6d9d700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64c81535f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f64bc011aa0 tx=0x7f64bc011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc00f8c0 con 0x7f64b0004400 2023-11-24T06:21:30.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc00fa20 con 0x7f64b0004400 2023-11-24T06:21:30.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c81607b0 con 0x7f64b0004400 2023-11-24T06:21:30.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc00fdd0 con 0x7f64b0004400 2023-11-24T06:21:30.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.484+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c8160d70 con 0x7f64b0004400 2023-11-24T06:21:30.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.485+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f64c80cff10 con 0x7f64b0004400 2023-11-24T06:21:30.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.485+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f64bc01d400 con 0x7f64b0004400 2023-11-24T06:21:30.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.486+0000 7f64af7fe700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64b406aff0 0x7f64b406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.486+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f64bc08a090 con 0x7f64b0004400 2023-11-24T06:21:30.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.486+0000 7f64c659c700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64b406aff0 0x7f64b406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.486+0000 7f64af7fe700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f64b4070e50 0x7f64b4073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.486+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 --> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64b4073940 con 0x7f64b4070e50 2023-11-24T06:21:30.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.486+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64bc08a4e0 con 0x7f64b0004400 2023-11-24T06:21:30.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.486+0000 7f64c759e700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f64b4070e50 0x7f64b4073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.486+0000 7f64c659c700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64b406aff0 0x7f64b406d4b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f64b8001930 tx=0x7f64b8007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.487+0000 7f64c759e700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f64b4070e50 0x7f64b4073270 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.488+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 <== osd.5 v2:172.21.15.123:6808/410207526 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f64b4073940 con 0x7f64b4070e50 2023-11-24T06:21:30.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.502+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 --> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f64c81627e0 con 0x7f64b4070e50 2023-11-24T06:21:30.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64af7fe700 1 -- 172.21.15.89:0/508268421 <== osd.5 v2:172.21.15.123:6808/410207526 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f64c0002080 con 0x7f64b4070e50 2023-11-24T06:21:30.506 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:30.507 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:21:30.507 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:30.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f64b4070e50 msgr2=0x7f64b4073270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.123:6808/410207526,v1:172.21.15.123:6809/410207526] conn(0x7f64b4070e50 0x7f64b4073270 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64b406aff0 msgr2=0x7f64b406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f64b406aff0 0x7f64b406d4b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f64b8001930 tx=0x7f64b8007000 comp rx=0 tx=0).stop 2023-11-24T06:21:30.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 msgr2=0x7f64c81535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 --2- 172.21.15.89:0/508268421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64b0004400 0x7f64c81535f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f64bc011aa0 tx=0x7f64bc011db0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64c759e700 1 -- 172.21.15.89:0/508268421 reap_dead start 2023-11-24T06:21:30.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 shutdown_connections 2023-11-24T06:21:30.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 >> 172.21.15.89:0/508268421 conn(0x7f64c8072710 msgr2=0x7f64c80ccd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:30.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 shutdown_connections 2023-11-24T06:21:30.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.503+0000 7f64cd19c700 1 -- 172.21.15.89:0/508268421 wait complete. 2023-11-24T06:21:30.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-11-24T06:21:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.636+0000 7efd5c911700 1 Processor -- start 2023-11-24T06:21:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.636+0000 7efd5c911700 1 -- start start 2023-11-24T06:21:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5c911700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efd5815ae20 0x7efd580b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5c911700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815b200 0x7efd580b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5c911700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd580cfd40 con 0x7efd580cf780 2023-11-24T06:21:30.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5c911700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd580d0020 con 0x7efd5815ae20 2023-11-24T06:21:30.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5c911700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd580affa0 con 0x7efd5815b200 2023-11-24T06:21:30.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5659c700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efd5815ae20 0x7efd580b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd56d9d700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efd580cf780 0x7efd580b0a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43374/0 (socket says 172.21.15.89:43374) 2023-11-24T06:21:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd56d9d700 1 -- 172.21.15.89:0/254210837 learned_addr learned my addr 172.21.15.89:0/254210837 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd55d9b700 1 --2- 172.21.15.89:0/254210837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815b200 0x7efd580b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5659c700 1 -- 172.21.15.89:0/254210837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815b200 msgr2=0x7efd580b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5659c700 1 --2- 172.21.15.89:0/254210837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815b200 0x7efd580b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5659c700 1 -- 172.21.15.89:0/254210837 >> v1:172.21.15.89:6789/0 conn(0x7efd580cf780 legacy=0x7efd580b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5659c700 1 -- 172.21.15.89:0/254210837 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd580b0280 con 0x7efd5815ae20 2023-11-24T06:21:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5659c700 1 --2- 172.21.15.89:0/254210837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd5815ae20 0x7efd580b29a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7efd48009840 tx=0x7efd48009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396b9c1729fbe37e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5559a700 1 -- 172.21.15.89:0/254210837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48011830 con 0x7efd5815ae20 2023-11-24T06:21:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5559a700 1 -- 172.21.15.89:0/254210837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd48011990 con 0x7efd5815ae20 2023-11-24T06:21:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.637+0000 7efd5559a700 1 -- 172.21.15.89:0/254210837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48011d20 con 0x7efd5815ae20 2023-11-24T06:21:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.638+0000 7efd5c911700 1 -- 172.21.15.89:0/254210837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd5815ae20 msgr2=0x7efd580b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.638+0000 7efd5c911700 1 --2- 172.21.15.89:0/254210837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd5815ae20 0x7efd580b29a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7efd48009840 tx=0x7efd48009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.638+0000 7efd5c911700 1 -- 172.21.15.89:0/254210837 shutdown_connections 2023-11-24T06:21:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.638+0000 7efd5c911700 1 --2- 172.21.15.89:0/254210837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd5815ae20 0x7efd580b29a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.638+0000 7efd5c911700 1 --2- 172.21.15.89:0/254210837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815b200 0x7efd580b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.638+0000 7efd5c911700 1 -- 172.21.15.89:0/254210837 >> 172.21.15.89:0/254210837 conn(0x7efd580b57b0 msgr2=0x7efd580b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.638+0000 7efd5c911700 1 -- 172.21.15.89:0/254210837 shutdown_connections 2023-11-24T06:21:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.638+0000 7efd5c911700 1 -- 172.21.15.89:0/254210837 wait complete. 2023-11-24T06:21:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.639+0000 7efd5c911700 1 Processor -- start 2023-11-24T06:21:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.639+0000 7efd5c911700 1 -- start start 2023-11-24T06:21:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5c911700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd580cf780 0x7efd5814f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5c911700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815ae20 0x7efd5809da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5c911700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd5815b200 0x7efd5809b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5c911700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5809d590 con 0x7efd580cf780 2023-11-24T06:21:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5c911700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5814f820 con 0x7efd5815b200 2023-11-24T06:21:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5c911700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5809a760 con 0x7efd5815ae20 2023-11-24T06:21:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd580cf780 0x7efd5814f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd580cf780 0x7efd5814f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39872/0 (socket says 172.21.15.89:39872) 2023-11-24T06:21:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd5659c700 1 -- 172.21.15.89:0/2755631223 learned_addr learned my addr 172.21.15.89:0/2755631223 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd55d9b700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815ae20 0x7efd5809da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.640+0000 7efd56d9d700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd5815b200 0x7efd5809b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd5659c700 1 -- 172.21.15.89:0/2755631223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815ae20 msgr2=0x7efd5809da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd5659c700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815ae20 0x7efd5809da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd5659c700 1 -- 172.21.15.89:0/2755631223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd5815b200 msgr2=0x7efd5809b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd5659c700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd5815b200 0x7efd5809b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd5659c700 1 -- 172.21.15.89:0/2755631223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd480094a0 con 0x7efd580cf780 2023-11-24T06:21:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd5659c700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd580cf780 0x7efd5814f2e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7efd48009b90 tx=0x7efd48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd46ffd700 1 -- 172.21.15.89:0/2755631223 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd480036a0 con 0x7efd580cf780 2023-11-24T06:21:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd46ffd700 1 -- 172.21.15.89:0/2755631223 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4801f070 con 0x7efd580cf780 2023-11-24T06:21:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5809aa40 con 0x7efd580cf780 2023-11-24T06:21:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd46ffd700 1 -- 172.21.15.89:0/2755631223 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48003a60 con 0x7efd580cf780 2023-11-24T06:21:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.641+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5809b000 con 0x7efd580cf780 2023-11-24T06:21:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.642+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd58067f60 con 0x7efd580cf780 2023-11-24T06:21:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.642+0000 7efd46ffd700 1 -- 172.21.15.89:0/2755631223 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7efd48003bc0 con 0x7efd580cf780 2023-11-24T06:21:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.643+0000 7efd46ffd700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd3806aff0 0x7efd3806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.643+0000 7efd46ffd700 1 -- 172.21.15.89:0/2755631223 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efd48021070 con 0x7efd580cf780 2023-11-24T06:21:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.643+0000 7efd55d9b700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd3806aff0 0x7efd3806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.643+0000 7efd55d9b700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd3806aff0 0x7efd3806d4b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efd5815aa80 tx=0x7efd4c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.647+0000 7efd46ffd700 1 -- 172.21.15.89:0/2755631223 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd48055ff0 con 0x7efd580cf780 2023-11-24T06:21:30.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.840+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7efd580b37d0 con 0x7efd3806aff0 2023-11-24T06:21:30.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.841+0000 7efd46ffd700 1 -- 172.21.15.89:0/2755631223 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7efd4c0076b0 con 0x7efd3806aff0 2023-11-24T06:21:30.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.842+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd3806aff0 msgr2=0x7efd3806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.842+0000 7efd5c911700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd3806aff0 0x7efd3806d4b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efd5815aa80 tx=0x7efd4c008040 comp rx=0 tx=0).stop 2023-11-24T06:21:30.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.842+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd580cf780 msgr2=0x7efd5814f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.842+0000 7efd5c911700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd580cf780 0x7efd5814f2e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7efd48009b90 tx=0x7efd48004750 comp rx=0 tx=0).stop 2023-11-24T06:21:30.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.843+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 shutdown_connections 2023-11-24T06:21:30.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.843+0000 7efd5c911700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7efd3806aff0 0x7efd3806d4b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.843+0000 7efd5c911700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd5815b200 0x7efd5809b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.843+0000 7efd5c911700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd580cf780 0x7efd5814f2e0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.843+0000 7efd5c911700 1 --2- 172.21.15.89:0/2755631223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd5815ae20 0x7efd5809da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.843+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 >> 172.21.15.89:0/2755631223 conn(0x7efd580b57b0 msgr2=0x7efd5806d400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:30.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.843+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 shutdown_connections 2023-11-24T06:21:30.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.843+0000 7efd5c911700 1 -- 172.21.15.89:0/2755631223 wait complete. 2023-11-24T06:21:30.851 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 5 to deep-scrub 2023-11-24T06:21:30.858 INFO:tasks.ceph:Scrubbing osd.6 2023-11-24T06:21:30.859 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:21:30.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.980+0000 7f46c1b4f700 1 Processor -- start 2023-11-24T06:21:30.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.980+0000 7f46c1b4f700 1 -- start start 2023-11-24T06:21:30.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46c1b4f700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46bc144600 0x7f46bc0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46c1b4f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 0x7f46bc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46c1b4f700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc06f1c0 con 0x7f46bc06f480 2023-11-24T06:21:30.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46c1b4f700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46bc06aff0 con 0x7f46bc144600 2023-11-24T06:21:30.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46c1b4f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc06c9b0 con 0x7f46bc06e830 2023-11-24T06:21:30.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bb7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46bc144600 0x7f46bc0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bb7fe700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46bc144600 0x7f46bc0b4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39880/0 (socket says 172.21.15.89:39880) 2023-11-24T06:21:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46baffd700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f46bc06f480 0x7f46bc06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43384/0 (socket says 172.21.15.89:43384) 2023-11-24T06:21:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bb7fe700 1 -- 172.21.15.89:0/1753151036 learned_addr learned my addr 172.21.15.89:0/1753151036 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bbfff700 1 --2- 172.21.15.89:0/1753151036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 0x7f46bc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bb7fe700 1 -- 172.21.15.89:0/1753151036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 msgr2=0x7f46bc06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bb7fe700 1 --2- 172.21.15.89:0/1753151036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 0x7f46bc06ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bb7fe700 1 -- 172.21.15.89:0/1753151036 >> v1:172.21.15.89:6789/0 conn(0x7f46bc06f480 legacy=0x7f46bc06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bb7fe700 1 -- 172.21.15.89:0/1753151036 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc06b200 con 0x7f46bc144600 2023-11-24T06:21:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46ba7fc700 1 -- 172.21.15.89:0/1753151036 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342573062 0 0) 0x7f46ac001110 con 0x7f46bc06f480 2023-11-24T06:21:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.981+0000 7f46bb7fe700 1 --2- 172.21.15.89:0/1753151036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 0x7f46bc0b4ee0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f46a40098f0 tx=0x7f46a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b378a6b91e33a1c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46ba7fc700 1 -- 172.21.15.89:0/1753151036 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a400b8a0 con 0x7f46bc144600 2023-11-24T06:21:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46ba7fc700 1 -- 172.21.15.89:0/1753151036 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a400ba00 con 0x7f46bc144600 2023-11-24T06:21:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46ba7fc700 1 -- 172.21.15.89:0/1753151036 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a400bd90 con 0x7f46bc144600 2023-11-24T06:21:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1753151036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 msgr2=0x7f46bc0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46c1b4f700 1 --2- 172.21.15.89:0/1753151036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 0x7f46bc0b4ee0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f46a40098f0 tx=0x7f46a4009c00 comp rx=0 tx=0).stop 2023-11-24T06:21:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1753151036 shutdown_connections 2023-11-24T06:21:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46c1b4f700 1 --2- 172.21.15.89:0/1753151036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 0x7f46bc06ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46c1b4f700 1 --2- 172.21.15.89:0/1753151036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 0x7f46bc0b4ee0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1753151036 >> 172.21.15.89:0/1753151036 conn(0x7f46bc0d0b40 msgr2=0x7f46bc0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:30.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1753151036 shutdown_connections 2023-11-24T06:21:30.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.982+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1753151036 wait complete. 2023-11-24T06:21:30.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.983+0000 7f46c1b4f700 1 Processor -- start 2023-11-24T06:21:30.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46c1b4f700 1 -- start start 2023-11-24T06:21:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46c1b4f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 0x7f46bc13f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46c1b4f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46bc06f480 0x7f46bc15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46c1b4f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 0x7f46bc073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46c1b4f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc15df60 con 0x7f46bc144600 2023-11-24T06:21:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46c1b4f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc15e240 con 0x7f46bc06f480 2023-11-24T06:21:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46c1b4f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc15e520 con 0x7f46bc06e830 2023-11-24T06:21:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46bbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 0x7f46bc073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46bbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 0x7f46bc073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39890/0 (socket says 172.21.15.89:39890) 2023-11-24T06:21:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.984+0000 7f46bbfff700 1 -- 172.21.15.89:0/1889150748 learned_addr learned my addr 172.21.15.89:0/1889150748 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46bb7fe700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 0x7f46bc13f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46baffd700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46bc06f480 0x7f46bc15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46bbfff700 1 -- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 msgr2=0x7f46bc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46bbfff700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46bc06e830 0x7f46bc13f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46bbfff700 1 -- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46bc06f480 msgr2=0x7f46bc15d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46bbfff700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46bc06f480 0x7f46bc15d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46bbfff700 1 -- 172.21.15.89:0/1889150748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a4009550 con 0x7f46bc144600 2023-11-24T06:21:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46baffd700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46bc06f480 0x7f46bc15d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46bbfff700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 0x7f46bc073f90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f46b400cc30 tx=0x7f46b400a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b400ab70 con 0x7f46bc144600 2023-11-24T06:21:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b400acd0 con 0x7f46bc144600 2023-11-24T06:21:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46bc15e800 con 0x7f46bc144600 2023-11-24T06:21:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4003c40 con 0x7f46bc144600 2023-11-24T06:21:30.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.985+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc15a630 con 0x7f46bc144600 2023-11-24T06:21:30.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.986+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f46bc15e3b0 con 0x7f46bc144600 2023-11-24T06:21:30.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.986+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f46b40213e0 con 0x7f46bc144600 2023-11-24T06:21:30.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46b3fff700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f469c06aff0 0x7f469c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f46b4089c20 con 0x7f46bc144600 2023-11-24T06:21:30.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46bb7fe700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f469c06aff0 0x7f469c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46b3fff700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7f469c070e50 0x7f469c073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:30.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 --> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f469c073940 con 0x7f469c070e50 2023-11-24T06:21:30.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46b408a070 con 0x7f46bc144600 2023-11-24T06:21:30.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46baffd700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7f469c070e50 0x7f469c073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:30.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46bb7fe700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f469c06aff0 0x7f469c06d4b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f46a4000c00 tx=0x7f46a4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.987+0000 7f46baffd700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7f469c070e50 0x7f469c073270 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:30.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:30.988+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 <== osd.6 v2:172.21.15.123:6824/668316489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f469c073940 con 0x7f469c070e50 2023-11-24T06:21:31.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.003+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 --> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f46bc06baa0 con 0x7f469c070e50 2023-11-24T06:21:31.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46b3fff700 1 -- 172.21.15.89:0/1889150748 <== osd.6 v2:172.21.15.123:6824/668316489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f46ac001440 con 0x7f469c070e50 2023-11-24T06:21:31.007 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:21:31.007 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:31.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7f469c070e50 msgr2=0x7f469c073270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.123:6824/668316489,v1:172.21.15.123:6825/668316489] conn(0x7f469c070e50 0x7f469c073270 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f469c06aff0 msgr2=0x7f469c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f469c06aff0 0x7f469c06d4b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f46a4000c00 tx=0x7f46a4008030 comp rx=0 tx=0).stop 2023-11-24T06:21:31.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 msgr2=0x7f46bc073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 --2- 172.21.15.89:0/1889150748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46bc144600 0x7f46bc073f90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f46b400cc30 tx=0x7f46b400a1f0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46bbfff700 1 -- 172.21.15.89:0/1889150748 reap_dead start 2023-11-24T06:21:31.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 shutdown_connections 2023-11-24T06:21:31.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 >> 172.21.15.89:0/1889150748 conn(0x7f46bc0d0b40 msgr2=0x7f46bc0b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:31.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 shutdown_connections 2023-11-24T06:21:31.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.004+0000 7f46c1b4f700 1 -- 172.21.15.89:0/1889150748 wait complete. 2023-11-24T06:21:31.019 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-11-24T06:21:31.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.136+0000 7fc326cf8700 1 Processor -- start 2023-11-24T06:21:31.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.136+0000 7fc326cf8700 1 -- start start 2023-11-24T06:21:31.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc326cf8700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc32015bdd0 0x7fc3201604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc326cf8700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc3200d1c70 0x7fc3200d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc326cf8700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3200b22d0 con 0x7fc3200b3370 2023-11-24T06:21:31.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc326cf8700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3200b2f50 con 0x7fc3200d1c70 2023-11-24T06:21:31.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc326cf8700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3200d2640 con 0x7fc32015bdd0 2023-11-24T06:21:31.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc325295700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc3200d1c70 0x7fc3200d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc325295700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc3200d1c70 0x7fc3200d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39900/0 (socket says 172.21.15.89:39900) 2023-11-24T06:21:31.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc325295700 1 -- 172.21.15.89:0/901714180 learned_addr learned my addr 172.21.15.89:0/901714180 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:31.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc324a94700 1 --2- 172.21.15.89:0/901714180 >> v2:172.21.15.123:3300/0 conn(0x7fc32015bdd0 0x7fc3201604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.137+0000 7fc325295700 1 -- 172.21.15.89:0/901714180 >> v2:172.21.15.123:3300/0 conn(0x7fc32015bdd0 msgr2=0x7fc3201604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:31.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc325295700 1 --2- 172.21.15.89:0/901714180 >> v2:172.21.15.123:3300/0 conn(0x7fc32015bdd0 0x7fc3201604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc325295700 1 -- 172.21.15.89:0/901714180 >> v1:172.21.15.123:6789/0 conn(0x7fc3200b3370 legacy=0x7fc3200b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:31.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc325295700 1 -- 172.21.15.89:0/901714180 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3200b25b0 con 0x7fc3200d1c70 2023-11-24T06:21:31.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc325295700 1 --2- 172.21.15.89:0/901714180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc3200d2100 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc32006ec10 tx=0x7fc31800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb753deb2ec7b44 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc31f7fe700 1 -- 172.21.15.89:0/901714180 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31800e070 con 0x7fc3200d1c70 2023-11-24T06:21:31.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc31f7fe700 1 -- 172.21.15.89:0/901714180 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318003a20 con 0x7fc3200d1c70 2023-11-24T06:21:31.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc31f7fe700 1 -- 172.21.15.89:0/901714180 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318003dd0 con 0x7fc3200d1c70 2023-11-24T06:21:31.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc326cf8700 1 -- 172.21.15.89:0/901714180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 msgr2=0x7fc3200d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc326cf8700 1 --2- 172.21.15.89:0/901714180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc3200d2100 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc32006ec10 tx=0x7fc31800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:31.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc326cf8700 1 -- 172.21.15.89:0/901714180 shutdown_connections 2023-11-24T06:21:31.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc326cf8700 1 --2- 172.21.15.89:0/901714180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc3200d2100 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc326cf8700 1 --2- 172.21.15.89:0/901714180 >> v2:172.21.15.123:3300/0 conn(0x7fc32015bdd0 0x7fc3201604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc326cf8700 1 -- 172.21.15.89:0/901714180 >> 172.21.15.89:0/901714180 conn(0x7fc3200731a0 msgr2=0x7fc32009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:31.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc326cf8700 1 -- 172.21.15.89:0/901714180 shutdown_connections 2023-11-24T06:21:31.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.138+0000 7fc326cf8700 1 -- 172.21.15.89:0/901714180 wait complete. 2023-11-24T06:21:31.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.140+0000 7fc326cf8700 1 Processor -- start 2023-11-24T06:21:31.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.140+0000 7fc326cf8700 1 -- start start 2023-11-24T06:21:31.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.140+0000 7fc326cf8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3200b3370 0x7fc3201573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.140+0000 7fc326cf8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc32015edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.140+0000 7fc326cf8700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc32015bdd0 0x7fc320159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.140+0000 7fc326cf8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200b55e0 con 0x7fc3200d1c70 2023-11-24T06:21:31.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.140+0000 7fc326cf8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200b49d0 con 0x7fc3200b3370 2023-11-24T06:21:31.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc326cf8700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200b4cb0 con 0x7fc32015bdd0 2023-11-24T06:21:31.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc32015edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc32015edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39904/0 (socket says 172.21.15.89:39904) 2023-11-24T06:21:31.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 -- 172.21.15.89:0/1993413891 learned_addr learned my addr 172.21.15.89:0/1993413891 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:31.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc324a94700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3200b3370 0x7fc3201573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc325295700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc32015bdd0 0x7fc320159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 -- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc32015bdd0 msgr2=0x7fc320159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc32015bdd0 0x7fc320159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 -- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3200b3370 msgr2=0x7fc3201573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3200b3370 0x7fc3201573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 -- 172.21.15.89:0/1993413891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc318009b30 con 0x7fc3200d1c70 2023-11-24T06:21:31.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc325295700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc32015bdd0 0x7fc320159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:31.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31ffff700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc32015edc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00c4a0 tx=0x7fc30c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31d7fa700 1 -- 172.21.15.89:0/1993413891 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c011790 con 0x7fc3200d1c70 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31d7fa700 1 -- 172.21.15.89:0/1993413891 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30c0118f0 con 0x7fc3200d1c70 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3200b4f90 con 0x7fc3200d1c70 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc31d7fa700 1 -- 172.21.15.89:0/1993413891 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c011ca0 con 0x7fc3200d1c70 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.141+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc320157930 con 0x7fc3200d1c70 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.142+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3200b5790 con 0x7fc3200d1c70 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.143+0000 7fc31d7fa700 1 -- 172.21.15.89:0/1993413891 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fc30c01c460 con 0x7fc3200d1c70 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.143+0000 7fc31d7fa700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc30406aff0 0x7fc30406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.143+0000 7fc31d7fa700 1 -- 172.21.15.89:0/1993413891 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc30c089120 con 0x7fc3200d1c70 2023-11-24T06:21:31.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.143+0000 7fc324a94700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc30406aff0 0x7fc30406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.144+0000 7fc324a94700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc30406aff0 0x7fc30406d4b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc314007a30 tx=0x7fc314008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.147+0000 7fc31d7fa700 1 -- 172.21.15.89:0/1993413891 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3200b5790 con 0x7fc3200d1c70 2023-11-24T06:21:31.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.342+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3200b2b70 con 0x7fc30406aff0 2023-11-24T06:21:31.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.342+0000 7fc31d7fa700 1 -- 172.21.15.89:0/1993413891 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc314004020 con 0x7fc30406aff0 2023-11-24T06:21:31.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc30406aff0 msgr2=0x7fc30406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc30406aff0 0x7fc30406d4b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc314007a30 tx=0x7fc314008040 comp rx=0 tx=0).stop 2023-11-24T06:21:31.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 msgr2=0x7fc32015edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc32015edc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00c4a0 tx=0x7fc30c00c7b0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 shutdown_connections 2023-11-24T06:21:31.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc32015bdd0 0x7fc320159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fc30406aff0 0x7fc30406d4b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3200b3370 0x7fc3201573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 --2- 172.21.15.89:0/1993413891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3200d1c70 0x7fc32015edc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 >> 172.21.15.89:0/1993413891 conn(0x7fc3200731a0 msgr2=0x7fc3200b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:31.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 shutdown_connections 2023-11-24T06:21:31.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.344+0000 7fc326cf8700 1 -- 172.21.15.89:0/1993413891 wait complete. 2023-11-24T06:21:31.350 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 6 to deep-scrub 2023-11-24T06:21:31.359 INFO:tasks.ceph:Scrubbing osd.7 2023-11-24T06:21:31.360 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:21:31.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.481+0000 7fd2ee51f700 1 Processor -- start 2023-11-24T06:21:31.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.481+0000 7fd2ee51f700 1 -- start start 2023-11-24T06:21:31.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2ee51f700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2e815a510 0x7fd2e80cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2ee51f700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80add20 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2ee51f700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80aeff0 con 0x7fd2e80ce3d0 2023-11-24T06:21:31.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2ee51f700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e80aebd0 con 0x7fd2e815a510 2023-11-24T06:21:31.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e7fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2e815a510 0x7fd2e80cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2ecabc700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd2e80ad8f0 0x7fd2e80afb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43416/0 (socket says 172.21.15.89:43416) 2023-11-24T06:21:31.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2ecabc700 1 -- 172.21.15.89:0/1889248404 learned_addr learned my addr 172.21.15.89:0/1889248404 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-11-24T06:21:31.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e7fff700 1 -- 172.21.15.89:0/1889248404 >> v1:172.21.15.123:6789/0 conn(0x7fd2e80ce3d0 legacy=0x7fd2e80ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e7fff700 1 -- 172.21.15.89:0/1889248404 >> v1:172.21.15.89:6789/0 conn(0x7fd2e80ad8f0 legacy=0x7fd2e80afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e7fff700 1 -- 172.21.15.89:0/1889248404 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80adf50 con 0x7fd2e815a510 2023-11-24T06:21:31.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e6ffd700 1 -- 172.21.15.89:0/1889248404 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892067795 0 0) 0x7fd2e80add20 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e7fff700 1 --2- 172.21.15.89:0/1889248404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e815a510 0x7fd2e80cde90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009840 tx=0x7fd2dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57a2d8fbaa54717 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e6ffd700 1 -- 172.21.15.89:0/1889248404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc011830 con 0x7fd2e815a510 2023-11-24T06:21:31.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e6ffd700 1 -- 172.21.15.89:0/1889248404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc011990 con 0x7fd2e815a510 2023-11-24T06:21:31.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.482+0000 7fd2e6ffd700 1 -- 172.21.15.89:0/1889248404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc011d20 con 0x7fd2e815a510 2023-11-24T06:21:31.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.483+0000 7fd2ee51f700 1 -- 172.21.15.89:0/1889248404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e815a510 msgr2=0x7fd2e80cde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.483+0000 7fd2ee51f700 1 --2- 172.21.15.89:0/1889248404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e815a510 0x7fd2e80cde90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009840 tx=0x7fd2dc009b50 comp rx=0 tx=0).stop 2023-11-24T06:21:31.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.483+0000 7fd2ee51f700 1 -- 172.21.15.89:0/1889248404 shutdown_connections 2023-11-24T06:21:31.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.483+0000 7fd2ee51f700 1 --2- 172.21.15.89:0/1889248404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e815a510 0x7fd2e80cde90 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.483+0000 7fd2ee51f700 1 -- 172.21.15.89:0/1889248404 >> 172.21.15.89:0/1889248404 conn(0x7fd2e8072710 msgr2=0x7fd2e8072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:31.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.483+0000 7fd2ee51f700 1 -- 172.21.15.89:0/1889248404 shutdown_connections 2023-11-24T06:21:31.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.483+0000 7fd2ee51f700 1 -- 172.21.15.89:0/1889248404 wait complete. 2023-11-24T06:21:31.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.484+0000 7fd2ee51f700 1 Processor -- start 2023-11-24T06:21:31.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.484+0000 7fd2ee51f700 1 -- start start 2023-11-24T06:21:31.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2ee51f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e80ad8f0 0x7fd2e80988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2ee51f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2e80ce3d0 0x7fd2e80c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2ee51f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e815a510 0x7fd2e8061a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2ee51f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80ca450 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2ee51f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e8098df0 con 0x7fd2e815a510 2023-11-24T06:21:31.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2ee51f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80c5380 con 0x7fd2e80ce3d0 2023-11-24T06:21:31.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e80ad8f0 0x7fd2e80988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e80ad8f0 0x7fd2e80988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39916/0 (socket says 172.21.15.89:39916) 2023-11-24T06:21:31.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e7fff700 1 -- 172.21.15.89:0/3220498886 learned_addr learned my addr 172.21.15.89:0/3220498886 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:31.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2ecabc700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e815a510 0x7fd2e8061a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e7fff700 1 -- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2e80ce3d0 msgr2=0x7fd2e80c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e77fe700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2e80ce3d0 0x7fd2e80c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e7fff700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2e80ce3d0 0x7fd2e80c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e7fff700 1 -- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e815a510 msgr2=0x7fd2e8061a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e7fff700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e815a510 0x7fd2e8061a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e7fff700 1 -- 172.21.15.89:0/3220498886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc0094a0 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2ecabc700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e815a510 0x7fd2e8061a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:31.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.485+0000 7fd2e77fe700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2e80ce3d0 0x7fd2e80c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:31.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.486+0000 7fd2e7fff700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e80ad8f0 0x7fd2e80988b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009b90 tx=0x7fd2dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.486+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc0036a0 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.486+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc01e070 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.486+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e815dc30 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.486+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc003a00 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.486+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e80ca070 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.486+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2e80ae7f0 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.487+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fd2dc003b60 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.488+0000 7fd2e4ff9700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd2d406aff0 0x7fd2d406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.488+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd2dc08ac20 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.488+0000 7fd2e77fe700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd2d406aff0 0x7fd2d406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.488+0000 7fd2e4ff9700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fd2d4070e50 0x7fd2d4073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.488+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 --> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2d4073940 con 0x7fd2d4070e50 2023-11-24T06:21:31.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.488+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2dc08b070 con 0x7fd2e80ad8f0 2023-11-24T06:21:31.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.488+0000 7fd2e77fe700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd2d406aff0 0x7fd2d406d4b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8153aa0 tx=0x7fd2d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.488+0000 7fd2ecabc700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fd2d4070e50 0x7fd2d4073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.489+0000 7fd2ecabc700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fd2d4070e50 0x7fd2d4073270 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.489+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 <== osd.7 v2:172.21.15.123:6800/751799151 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd2d4073940 con 0x7fd2d4070e50 2023-11-24T06:21:31.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.504+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 --> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd2e80ae510 con 0x7fd2d4070e50 2023-11-24T06:21:31.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2e4ff9700 1 -- 172.21.15.89:0/3220498886 <== osd.7 v2:172.21.15.123:6800/751799151 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd2e00070e0 con 0x7fd2d4070e50 2023-11-24T06:21:31.508 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-11-24T06:21:31.508 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:21:31.508 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-11-24T06:21:31.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fd2d4070e50 msgr2=0x7fd2d4073270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.123:6800/751799151,v1:172.21.15.123:6801/751799151] conn(0x7fd2d4070e50 0x7fd2d4073270 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd2d406aff0 msgr2=0x7fd2d406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fd2d406aff0 0x7fd2d406d4b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8153aa0 tx=0x7fd2d8009040 comp rx=0 tx=0).stop 2023-11-24T06:21:31.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e80ad8f0 msgr2=0x7fd2e80988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 --2- 172.21.15.89:0/3220498886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e80ad8f0 0x7fd2e80988b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009b90 tx=0x7fd2dc004750 comp rx=0 tx=0).stop 2023-11-24T06:21:31.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ecabc700 1 -- 172.21.15.89:0/3220498886 reap_dead start 2023-11-24T06:21:31.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 shutdown_connections 2023-11-24T06:21:31.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 >> 172.21.15.89:0/3220498886 conn(0x7fd2e8072710 msgr2=0x7fd2e814fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:31.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 shutdown_connections 2023-11-24T06:21:31.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.505+0000 7fd2ee51f700 1 -- 172.21.15.89:0/3220498886 wait complete. 2023-11-24T06:21:31.520 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-11-24T06:21:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.637+0000 7fdcf9878700 1 Processor -- start 2023-11-24T06:21:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.637+0000 7fdcf9878700 1 -- start start 2023-11-24T06:21:31.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.638+0000 7fdcf9878700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdcf415a2e0 0x7fdcf40ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.638+0000 7fdcf9878700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf4071a50 0x7fdcf40b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.638+0000 7fdcf9878700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf415ed40 con 0x7fdcf406ec00 2023-11-24T06:21:31.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.638+0000 7fdcf9878700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf40b42a0 con 0x7fdcf415a2e0 2023-11-24T06:21:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.638+0000 7fdcf9878700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf40b58e0 con 0x7fdcf4071a50 2023-11-24T06:21:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf2ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdcf415a2e0 0x7fdcf40ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf2ffd700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdcf415a2e0 0x7fdcf40ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39926/0 (socket says 172.21.15.89:39926) 2023-11-24T06:21:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf2ffd700 1 -- 172.21.15.89:0/613106478 learned_addr learned my addr 172.21.15.89:0/613106478 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf37fe700 1 --2- 172.21.15.89:0/613106478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf4071a50 0x7fdcf40b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf2ffd700 1 -- 172.21.15.89:0/613106478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf4071a50 msgr2=0x7fdcf40b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:31.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf2ffd700 1 --2- 172.21.15.89:0/613106478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf4071a50 0x7fdcf40b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf2ffd700 1 -- 172.21.15.89:0/613106478 >> v1:172.21.15.123:6789/0 conn(0x7fdcf406ec00 legacy=0x7fdcf4071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf2ffd700 1 -- 172.21.15.89:0/613106478 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf415dee0 con 0x7fdcf415a2e0 2023-11-24T06:21:31.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf2ffd700 1 --2- 172.21.15.89:0/613106478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ce000 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fdce400bb90 tx=0x7fdce400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802e49a23d117dda server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf1ffb700 1 -- 172.21.15.89:0/613106478 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400dd50 con 0x7fdcf415a2e0 2023-11-24T06:21:31.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf1ffb700 1 -- 172.21.15.89:0/613106478 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce4010470 con 0x7fdcf415a2e0 2023-11-24T06:21:31.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.639+0000 7fdcf1ffb700 1 -- 172.21.15.89:0/613106478 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4010800 con 0x7fdcf415a2e0 2023-11-24T06:21:31.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.640+0000 7fdcf9878700 1 -- 172.21.15.89:0/613106478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 msgr2=0x7fdcf40ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.640+0000 7fdcf9878700 1 --2- 172.21.15.89:0/613106478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ce000 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fdce400bb90 tx=0x7fdce400bea0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.640+0000 7fdcf9878700 1 -- 172.21.15.89:0/613106478 shutdown_connections 2023-11-24T06:21:31.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.640+0000 7fdcf9878700 1 --2- 172.21.15.89:0/613106478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf4071a50 0x7fdcf40b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.640+0000 7fdcf9878700 1 --2- 172.21.15.89:0/613106478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ce000 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.640+0000 7fdcf9878700 1 -- 172.21.15.89:0/613106478 >> 172.21.15.89:0/613106478 conn(0x7fdcf40743b0 msgr2=0x7fdcf406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.640+0000 7fdcf9878700 1 -- 172.21.15.89:0/613106478 shutdown_connections 2023-11-24T06:21:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.640+0000 7fdcf9878700 1 -- 172.21.15.89:0/613106478 wait complete. 2023-11-24T06:21:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.641+0000 7fdcf9878700 1 Processor -- start 2023-11-24T06:21:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.641+0000 7fdcf9878700 1 -- start start 2023-11-24T06:21:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf9878700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf406ec00 0x7fdcf415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf9878700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcf4071a50 0x7fdcf40b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf9878700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf9878700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf40aa380 con 0x7fdcf415a2e0 2023-11-24T06:21:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf9878700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf40a9770 con 0x7fdcf4071a50 2023-11-24T06:21:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf9878700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf40a9a50 con 0x7fdcf406ec00 2023-11-24T06:21:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ac5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39936/0 (socket says 172.21.15.89:39936) 2023-11-24T06:21:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 -- 172.21.15.89:0/1250342485 learned_addr learned my addr 172.21.15.89:0/1250342485 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf27fc700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcf4071a50 0x7fdcf40b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf2ffd700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf406ec00 0x7fdcf415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 -- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf406ec00 msgr2=0x7fdcf415cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf406ec00 0x7fdcf415cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 -- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcf4071a50 msgr2=0x7fdcf40b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcf4071a50 0x7fdcf40b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 -- 172.21.15.89:0/1250342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce400b7f0 con 0x7fdcf415a2e0 2023-11-24T06:21:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf27fc700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcf4071a50 0x7fdcf40b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf2ffd700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf406ec00 0x7fdcf415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.642+0000 7fdcf37fe700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ac5e0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fdcec007e30 tx=0x7fdcec00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.643+0000 7fdcdbfff700 1 -- 172.21.15.89:0/1250342485 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec00eea0 con 0x7fdcf415a2e0 2023-11-24T06:21:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.643+0000 7fdcdbfff700 1 -- 172.21.15.89:0/1250342485 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcec014420 con 0x7fdcf415a2e0 2023-11-24T06:21:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.643+0000 7fdcdbfff700 1 -- 172.21.15.89:0/1250342485 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec0147d0 con 0x7fdcf415a2e0 2023-11-24T06:21:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.643+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf40a9d30 con 0x7fdcf415a2e0 2023-11-24T06:21:31.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.643+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf40a9460 con 0x7fdcf415a2e0 2023-11-24T06:21:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.644+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf40753f0 con 0x7fdcf415a2e0 2023-11-24T06:21:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.644+0000 7fdcdbfff700 1 -- 172.21.15.89:0/1250342485 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fdcec014930 con 0x7fdcf415a2e0 2023-11-24T06:21:31.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.644+0000 7fdcdbfff700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdce006aff0 0x7fdce006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.645+0000 7fdcdbfff700 1 -- 172.21.15.89:0/1250342485 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdcec089bd0 con 0x7fdcf415a2e0 2023-11-24T06:21:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.645+0000 7fdcf2ffd700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdce006aff0 0x7fdce006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.645+0000 7fdcf2ffd700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdce006aff0 0x7fdce006d4b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdcf415d2f0 tx=0x7fdce4013000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.649+0000 7fdcdbfff700 1 -- 172.21.15.89:0/1250342485 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdcec054e80 con 0x7fdcf415a2e0 2023-11-24T06:21:31.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.840+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcf415e6f0 con 0x7fdce006aff0 2023-11-24T06:21:31.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.840+0000 7fdcdbfff700 1 -- 172.21.15.89:0/1250342485 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdce400de50 con 0x7fdce006aff0 2023-11-24T06:21:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdce006aff0 msgr2=0x7fdce006d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdce006aff0 0x7fdce006d4b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdcf415d2f0 tx=0x7fdce4013000 comp rx=0 tx=0).stop 2023-11-24T06:21:31.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 msgr2=0x7fdcf40ac5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ac5e0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fdcec007e30 tx=0x7fdcec00e3c0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 shutdown_connections 2023-11-24T06:21:31.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcf415a2e0 0x7fdcf40ac5e0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdcf406ec00 0x7fdcf415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fdce006aff0 0x7fdce006d4b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 --2- 172.21.15.89:0/1250342485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcf4071a50 0x7fdcf40b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 >> 172.21.15.89:0/1250342485 conn(0x7fdcf40743b0 msgr2=0x7fdcf40b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:31.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.842+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 shutdown_connections 2023-11-24T06:21:31.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.843+0000 7fdcf9878700 1 -- 172.21.15.89:0/1250342485 wait complete. 2023-11-24T06:21:31.847 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 7 to deep-scrub 2023-11-24T06:21:31.857 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.974+0000 7f78eb239700 1 Processor -- start 2023-11-24T06:21:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.974+0000 7f78eb239700 1 -- start start 2023-11-24T06:21:31.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.975+0000 7f78eb239700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78e40d1b00 0x7f78e40d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.975+0000 7f78eb239700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e40ce4a0 con 0x7f78e40b1cb0 2023-11-24T06:21:31.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.975+0000 7f78eb239700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e40d2490 con 0x7f78e415e110 2023-11-24T06:21:31.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.975+0000 7f78eb239700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e40cefe0 con 0x7f78e40d1b00 2023-11-24T06:21:31.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.975+0000 7f78e3fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78e40d1b00 0x7f78e40d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.975+0000 7f78e3fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78e40d1b00 0x7f78e40d1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39944/0 (socket says 172.21.15.89:39944) 2023-11-24T06:21:31.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.975+0000 7f78e3fff700 1 -- 172.21.15.89:0/3588224152 learned_addr learned my addr 172.21.15.89:0/3588224152 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:31.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.975+0000 7f78e97d6700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f78e40b1cb0 0x7f78e40b2090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43446/0 (socket says 172.21.15.89:43446) 2023-11-24T06:21:31.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78e3fff700 1 -- 172.21.15.89:0/3588224152 >> v1:172.21.15.123:6789/0 conn(0x7f78e415e110 legacy=0x7f78e40b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:31.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78e3fff700 1 -- 172.21.15.89:0/3588224152 >> v1:172.21.15.89:6789/0 conn(0x7f78e40b1cb0 legacy=0x7f78e40b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78e3fff700 1 -- 172.21.15.89:0/3588224152 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e40cf470 con 0x7f78e40d1b00 2023-11-24T06:21:31.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78e37fe700 1 -- 172.21.15.89:0/3588224152 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870936031 0 0) 0x7f78e40ce4a0 con 0x7f78e40b1cb0 2023-11-24T06:21:31.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78e3fff700 1 --2- 172.21.15.89:0/3588224152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40d1f50 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f78d00098f0 tx=0x7f78d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba54631a0b1dbfd5 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78e37fe700 1 -- 172.21.15.89:0/3588224152 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d0004230 con 0x7f78e40d1b00 2023-11-24T06:21:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78e37fe700 1 -- 172.21.15.89:0/3588224152 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d0004390 con 0x7f78e40d1b00 2023-11-24T06:21:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78e37fe700 1 -- 172.21.15.89:0/3588224152 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d00036a0 con 0x7f78e40d1b00 2023-11-24T06:21:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78eb239700 1 -- 172.21.15.89:0/3588224152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 msgr2=0x7f78e40d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78eb239700 1 --2- 172.21.15.89:0/3588224152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40d1f50 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f78d00098f0 tx=0x7f78d0009c00 comp rx=0 tx=0).stop 2023-11-24T06:21:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78eb239700 1 -- 172.21.15.89:0/3588224152 shutdown_connections 2023-11-24T06:21:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78eb239700 1 --2- 172.21.15.89:0/3588224152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40d1f50 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.976+0000 7f78eb239700 1 -- 172.21.15.89:0/3588224152 >> 172.21.15.89:0/3588224152 conn(0x7f78e40730a0 msgr2=0x7f78e409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.977+0000 7f78eb239700 1 -- 172.21.15.89:0/3588224152 shutdown_connections 2023-11-24T06:21:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.977+0000 7f78eb239700 1 -- 172.21.15.89:0/3588224152 wait complete. 2023-11-24T06:21:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.978+0000 7f78eb239700 1 Processor -- start 2023-11-24T06:21:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.978+0000 7f78eb239700 1 -- start start 2023-11-24T06:21:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78eb239700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e40b1cb0 0x7f78e415e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78eb239700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78eb239700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78e415e110 0x7f78e40cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78eb239700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e4160e10 con 0x7f78e40d1b00 2023-11-24T06:21:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78eb239700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e40b54f0 con 0x7f78e40b1cb0 2023-11-24T06:21:31.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78eb239700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e40b4920 con 0x7f78e415e110 2023-11-24T06:21:31.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39948/0 (socket says 172.21.15.89:39948) 2023-11-24T06:21:31.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 -- 172.21.15.89:0/457723070 learned_addr learned my addr 172.21.15.89:0/457723070 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:31.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 -- 172.21.15.89:0/457723070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78e415e110 msgr2=0x7f78e40cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78e415e110 0x7f78e40cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 -- 172.21.15.89:0/457723070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e40b1cb0 msgr2=0x7f78e415e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:31.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e8fd5700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e40b1cb0 0x7f78e415e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e40b1cb0 0x7f78e415e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:31.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 -- 172.21.15.89:0/457723070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0009550 con 0x7f78e40d1b00 2023-11-24T06:21:31.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e3fff700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40ca540 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f78e4072df0 tx=0x7f78d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e17fa700 1 -- 172.21.15.89:0/457723070 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d0003dc0 con 0x7f78e40d1b00 2023-11-24T06:21:31.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e17fa700 1 -- 172.21.15.89:0/457723070 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d001f070 con 0x7f78e40d1b00 2023-11-24T06:21:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e40b4bc0 con 0x7f78e40d1b00 2023-11-24T06:21:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78e17fa700 1 -- 172.21.15.89:0/457723070 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d000b8a0 con 0x7f78e40d1b00 2023-11-24T06:21:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.979+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e40b5180 con 0x7f78e40d1b00 2023-11-24T06:21:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.980+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e40b5680 con 0x7f78e40d1b00 2023-11-24T06:21:31.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.981+0000 7f78e17fa700 1 -- 172.21.15.89:0/457723070 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f78d0021030 con 0x7f78e40d1b00 2023-11-24T06:21:31.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.981+0000 7f78e17fa700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f78c806aff0 0x7f78c806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:31.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.981+0000 7f78e8fd5700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f78c806aff0 0x7f78c806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:31.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.981+0000 7f78e17fa700 1 -- 172.21.15.89:0/457723070 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f78d008a980 con 0x7f78e40d1b00 2023-11-24T06:21:31.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.982+0000 7f78e8fd5700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f78c806aff0 0x7f78c806d4b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f78d800a7d0 tx=0x7f78d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:31.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:31.986+0000 7f78e17fa700 1 -- 172.21.15.89:0/457723070 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78d0055cb0 con 0x7f78e40d1b00 2023-11-24T06:21:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.174+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f78e40cec70 con 0x7f78c806aff0 2023-11-24T06:21:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.176+0000 7f78e17fa700 1 -- 172.21.15.89:0/457723070 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50820 (secure 0 0 0) 0x7f78d80113d0 con 0x7f78c806aff0 2023-11-24T06:21:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f78c806aff0 msgr2=0x7f78c806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f78c806aff0 0x7f78c806d4b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f78d800a7d0 tx=0x7f78d8008040 comp rx=0 tx=0).stop 2023-11-24T06:21:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 msgr2=0x7f78e40ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40ca540 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f78e4072df0 tx=0x7f78d0004750 comp rx=0 tx=0).stop 2023-11-24T06:21:32.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 shutdown_connections 2023-11-24T06:21:32.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78e415e110 0x7f78e40cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:32.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f78c806aff0 0x7f78c806d4b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:32.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e40b1cb0 0x7f78e415e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:32.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 --2- 172.21.15.89:0/457723070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40d1b00 0x7f78e40ca540 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:32.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 >> 172.21.15.89:0/457723070 conn(0x7f78e40730a0 msgr2=0x7f78e40cc220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:32.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 shutdown_connections 2023-11-24T06:21:32.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:32.178+0000 7f78eb239700 1 -- 172.21.15.89:0/457723070 wait complete. 2023-11-24T06:21:32.185 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:32.193 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2023-11-24T06:21:31.751072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":223764,"kb_used_data":7592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754750956,"statfs":{"total":773094113280,"available":772864978944,"internally_reserved":0,"allocated":7774208,"data_stored":3990872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-129,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.314673"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-24T06:21:22.435599+0000","last_change":"2023-11-24T06:18:38.879980+0000","last_active":"2023-11-24T06:21:22.435599+0000","last_peered":"2023-11-24T06:21:22.435599+0000","last_clean":"2023-11-24T06:21:22.435599+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:21:22.435599+0000","last_undegraded":"2023-11-24T06:21:22.435599+0000","last_fullsized":"2023-11-24T06:21:22.435599+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:18:37.811950+0000","last_clean_scrub_stamp":"2023-11-24T06:18:37.811950+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:49:08.706520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":1417216,"data_stored":941635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.47299999999999998},"min":{"1min":0.39800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.34200000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.61499999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.47999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65900000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.75,"15min":0.75},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.497},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.55600000000000005}]},{"osd":4,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.40000000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.64800000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.628,"15min":0.622},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.63300000000000001},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78000000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.436}]}]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.54500000000000004}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.64700000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63400000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.61499999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.35099999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.376}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.39800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.25},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53100000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.49199999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.995,"15min":0.995},"last":0.41999999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58999999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.39700000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69299999999999995,"5min":1.087,"15min":1.087},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.75700000000000001,"5min":1.161,"15min":1.161},"last":0.47399999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78400000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.66200000000000003},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82699999999999996,"5min":1.403,"15min":1.403},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.624,"15min":0.63200000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76000000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.206,"15min":1.206},"last":0.52900000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66600000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.79500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.621}]}]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":1417216,"data_stored":941635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.32100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.48399999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.312}]},{"osd":2,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47099999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.35999999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42599999999999999,"5min":0.626,"15min":0.626},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39100000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.5,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.34799999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.433}]},{"osd":5,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.46200000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.42099999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73099999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.47499999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88900000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.48899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34899999999999998,"15min":0.35499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.435},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54500000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.41199999999999998,"15min":0.41799999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56799999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.497}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60999999999999999,"5min":0.629,"15min":0.629},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59599999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.65000000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85399999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.622}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.71199999999999997,"15min":0.72999999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89100000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.92700000000000005,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0629999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.874,"5min":2.149,"15min":2.149},"last":0.70299999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97399999999999998,"5min":1.246,"15min":1.246},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.63500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.435,"5min":0.496,"15min":0.496},"last":0.248},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34499999999999997,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.39800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.27400000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.34200000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.376,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.33500000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46100000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.42399999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.32200000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.74399999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80900000000000005,"5min":0.874,"15min":0.874},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.624,"15min":0.624},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.75,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.62},"min":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78300000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.624},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.67300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93500000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.82999999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.67400000000000004,"15min":0.69599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71799999999999997,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.68600000000000005}]},{"osd":1,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.754},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.76700000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71299999999999997,"5min":0.876,"15min":0.876},"last":0.65100000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81499999999999995,"5min":0.879,"15min":0.879},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72799999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.746,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76700000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri Nov 24 06:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72399999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.872,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.66700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91900000000000004,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":3.04,"15min":3.04},"last":0.44400000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.39800000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.871,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":1.087,"15min":1.087},"last":0.502},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.95099999999999996,"15min":1.0129999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79900000000000004,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0009999999999999,"15min":1.0720000000000001},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":10.198,"15min":10.198},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.02,"15min":1.0960000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.71199999999999997,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.41099999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.33500000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36599999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.34999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:32.200 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-24T06:18:37.811950+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=6, tm_min=18, tm_sec=37, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=6, tm_min=21, tm_sec=26, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T06:21:32.201 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-24T06:21:48.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:21:48.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:21:52.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:21:52.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.333+0000 7fa80efa7700 1 Processor -- start 2023-11-24T06:21:52.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.333+0000 7fa80efa7700 1 -- start start 2023-11-24T06:21:52.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.334+0000 7fa80efa7700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa80815c030 0x7fa8080d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.334+0000 7fa80efa7700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8080ae910 0x7fa8080d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.334+0000 7fa80efa7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8080cd5d0 con 0x7fa8080cc8c0 2023-11-24T06:21:52.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.334+0000 7fa80efa7700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8080cd870 con 0x7fa8080ae910 2023-11-24T06:21:52.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.334+0000 7fa80efa7700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8080cdb50 con 0x7fa80815c030 2023-11-24T06:21:52.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa807fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8080ae910 0x7fa8080d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa807fff700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8080ae910 0x7fa8080d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39954/0 (socket says 172.21.15.89:39954) 2023-11-24T06:21:52.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa807fff700 1 -- 172.21.15.89:0/2774368945 learned_addr learned my addr 172.21.15.89:0/2774368945 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:52.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa80cd43700 1 --2- 172.21.15.89:0/2774368945 >> v2:172.21.15.123:3300/0 conn(0x7fa80815c030 0x7fa8080d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa807fff700 1 -- 172.21.15.89:0/2774368945 >> v2:172.21.15.123:3300/0 conn(0x7fa80815c030 msgr2=0x7fa8080d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:52.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa807fff700 1 --2- 172.21.15.89:0/2774368945 >> v2:172.21.15.123:3300/0 conn(0x7fa80815c030 0x7fa8080d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa807fff700 1 -- 172.21.15.89:0/2774368945 >> v1:172.21.15.123:6789/0 conn(0x7fa8080cc8c0 legacy=0x7fa8080ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:52.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa807fff700 1 -- 172.21.15.89:0/2774368945 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8080cde30 con 0x7fa8080ae910 2023-11-24T06:21:52.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa807fff700 1 --2- 172.21.15.89:0/2774368945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8080ae910 0x7fa8080d05b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa808073610 tx=0x7fa7f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932a30caa12c1c88 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa8077fe700 1 -- 172.21.15.89:0/2774368945 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4004230 con 0x7fa8080ae910 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa8077fe700 1 -- 172.21.15.89:0/2774368945 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f4004390 con 0x7fa8080ae910 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.335+0000 7fa8077fe700 1 -- 172.21.15.89:0/2774368945 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40036a0 con 0x7fa8080ae910 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.336+0000 7fa80efa7700 1 -- 172.21.15.89:0/2774368945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8080ae910 msgr2=0x7fa8080d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.336+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2774368945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8080ae910 0x7fa8080d05b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa808073610 tx=0x7fa7f4009bd0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.336+0000 7fa80efa7700 1 -- 172.21.15.89:0/2774368945 shutdown_connections 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.336+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2774368945 >> v2:172.21.15.123:3300/0 conn(0x7fa80815c030 0x7fa8080d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.336+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2774368945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8080ae910 0x7fa8080d05b0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.336+0000 7fa80efa7700 1 -- 172.21.15.89:0/2774368945 >> 172.21.15.89:0/2774368945 conn(0x7fa8080730a0 msgr2=0x7fa80809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:52.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.336+0000 7fa80efa7700 1 -- 172.21.15.89:0/2774368945 shutdown_connections 2023-11-24T06:21:52.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.336+0000 7fa80efa7700 1 -- 172.21.15.89:0/2774368945 wait complete. 2023-11-24T06:21:52.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.337+0000 7fa80efa7700 1 Processor -- start 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.337+0000 7fa80efa7700 1 -- start start 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80efa7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8080ae910 0x7fa8080c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80efa7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8080cc8c0 0x7fa8080c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80efa7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa80806c7c0 0x7fa8080c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80efa7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80806d020 con 0x7fa80806c7c0 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80efa7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080c4e50 con 0x7fa8080cc8c0 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80efa7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080c5e30 con 0x7fa8080ae910 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80d544700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa80806c7c0 0x7fa8080c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80d544700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa80806c7c0 0x7fa8080c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39964/0 (socket says 172.21.15.89:39964) 2023-11-24T06:21:52.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80d544700 1 -- 172.21.15.89:0/2760337182 learned_addr learned my addr 172.21.15.89:0/2760337182 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:52.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa807fff700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8080cc8c0 0x7fa8080c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80cd43700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8080ae910 0x7fa8080c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80d544700 1 -- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8080ae910 msgr2=0x7fa8080c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80d544700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8080ae910 0x7fa8080c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80d544700 1 -- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8080cc8c0 msgr2=0x7fa8080c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80d544700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8080cc8c0 0x7fa8080c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa80d544700 1 -- 172.21.15.89:0/2760337182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f4009550 con 0x7fa80806c7c0 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.338+0000 7fa807fff700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8080cc8c0 0x7fa8080c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.339+0000 7fa80d544700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa80806c7c0 0x7fa8080c4910 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa80000cc10 tx=0x7fa80000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.339+0000 7fa8057fa700 1 -- 172.21.15.89:0/2760337182 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80000abb0 con 0x7fa80806c7c0 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.339+0000 7fa8057fa700 1 -- 172.21.15.89:0/2760337182 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80000ad10 con 0x7fa80806c7c0 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.339+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8080c6110 con 0x7fa80806c7c0 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.339+0000 7fa8057fa700 1 -- 172.21.15.89:0/2760337182 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800012660 con 0x7fa80806c7c0 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.339+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8080c66d0 con 0x7fa80806c7c0 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.340+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa808069000 con 0x7fa80806c7c0 2023-11-24T06:21:52.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.340+0000 7fa8057fa700 1 -- 172.21.15.89:0/2760337182 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7fa8000127c0 con 0x7fa80806c7c0 2023-11-24T06:21:52.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.341+0000 7fa8057fa700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7ec06aff0 0x7fa7ec06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.341+0000 7fa8057fa700 1 -- 172.21.15.89:0/2760337182 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa800089cd0 con 0x7fa80806c7c0 2023-11-24T06:21:52.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.341+0000 7fa80cd43700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7ec06aff0 0x7fa7ec06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.341+0000 7fa80cd43700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7ec06aff0 0x7fa7ec06d4b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00b9b0 tx=0x7fa7fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:52.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.345+0000 7fa8057fa700 1 -- 172.21.15.89:0/2760337182 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa800054ee0 con 0x7fa80806c7c0 2023-11-24T06:21:52.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.531+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 --> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8080751e0 con 0x7fa7ec06aff0 2023-11-24T06:21:52.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.533+0000 7fa8057fa700 1 -- 172.21.15.89:0/2760337182 <== mgr.4101 v2:172.21.15.89:6832/2291342889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50773 (secure 0 0 0) 0x7fa8080751e0 con 0x7fa7ec06aff0 2023-11-24T06:21:52.537 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.535+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7ec06aff0 msgr2=0x7fa7ec06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.535+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7ec06aff0 0x7fa7ec06d4b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00b9b0 tx=0x7fa7fc008040 comp rx=0 tx=0).stop 2023-11-24T06:21:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.535+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa80806c7c0 msgr2=0x7fa8080c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.535+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa80806c7c0 0x7fa8080c4910 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa80000cc10 tx=0x7fa80000a060 comp rx=0 tx=0).stop 2023-11-24T06:21:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.536+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 shutdown_connections 2023-11-24T06:21:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.536+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa80806c7c0 0x7fa8080c4910 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.536+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8080ae910 0x7fa8080c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.536+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7fa7ec06aff0 0x7fa7ec06d4b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.536+0000 7fa80efa7700 1 --2- 172.21.15.89:0/2760337182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8080cc8c0 0x7fa8080c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.536+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 >> 172.21.15.89:0/2760337182 conn(0x7fa8080730a0 msgr2=0x7fa8080b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.536+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 shutdown_connections 2023-11-24T06:21:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.536+0000 7fa80efa7700 1 -- 172.21.15.89:0/2760337182 wait complete. 2023-11-24T06:21:52.540 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-11-24T06:21:52.551 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2023-11-24T06:21:51.755139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":223756,"kb_used_data":7592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754750964,"statfs":{"total":773094113280,"available":772864987136,"internally_reserved":0,"allocated":7774208,"data_stored":3990872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002321"},"pg_stats":[{"pgid":"1.0","version":"67'119","reported_seq":223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-24T06:21:35.379906+0000","last_change":"2023-11-24T06:21:35.379954+0000","last_active":"2023-11-24T06:21:35.379906+0000","last_peered":"2023-11-24T06:21:35.379906+0000","last_clean":"2023-11-24T06:21:35.379906+0000","last_became_active":"2023-11-24T06:18:38.878955+0000","last_became_peered":"2023-11-24T06:18:38.878955+0000","last_unstale":"2023-11-24T06:21:35.379906+0000","last_undegraded":"2023-11-24T06:21:35.379906+0000","last_fullsized":"2023-11-24T06:21:35.379906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-11-24T06:21:35.379789+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-11-24T06:21:35.379789+0000","last_clean_scrub_stamp":"2023-11-24T06:21:35.379789+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:31:21.092454+0000","scrub_duration":0.0060404650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28400,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343440,"statfs":{"total":96636764160,"available":96607682560,"internally_reserved":0,"allocated":1417216,"data_stored":941635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1587,"internal_metadata":27654605},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72599999999999998,"5min":0.751,"15min":0.751},"last":0.501},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67800000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.40000000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67300000000000004,"5min":0.749,"15min":0.749},"last":0.48799999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.40799999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.441,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71499999999999997,"5min":0.75,"15min":0.75},"last":0.45600000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40699999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77800000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54200000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.35099999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69599999999999995,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.36499999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58499999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61399999999999999,"15min":0.628},"min":{"1min":0.39100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73699999999999999,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.39300000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.627,"15min":0.621},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.44},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.623,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.748,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.42499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.83599999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0289999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.64100000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.78500000000000003,"15min":0.83099999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86099999999999999,"5min":6.423,"15min":6.423},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.83299999999999996,"15min":0.88100000000000001},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94999999999999996,"5min":7.4699999999999998,"15min":7.4699999999999998},"last":0.66000000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.64500000000000002,"15min":0.66100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93000000000000005,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.39300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.67100000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.097,"15min":1.1850000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":12.554,"15min":12.554},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.45000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.96599999999999997,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.82799999999999996},"min":{"1min":0.33000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"last":0.376}]},{"osd":5,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.60799999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:20:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.88,"15min":0.91900000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":6.1280000000000001,"15min":6.1280000000000001},"last":0.502},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.61899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32700000000000001,"15min":0.32900000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.436,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52800000000000002,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33400000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.34899999999999998,"15min":0.35699999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39900000000000002,"15min":0.41099999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.504,"5min":0.995,"15min":0.995},"last":0.35899999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40999999999999998,"15min":0.42199999999999999},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.55500000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.46000000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.38700000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58999999999999997,"15min":0.60599999999999998},"min":{"1min":0.40799999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.75600000000000001,"5min":1.087,"15min":1.087},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71199999999999997,"5min":1.161,"15min":1.161},"last":0.51000000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61099999999999999,"15min":0.626},"min":{"1min":0.47399999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80300000000000005,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.48599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81699999999999995,"5min":1.403,"15min":1.403},"last":0.56000000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61599999999999999,"15min":0.629},"min":{"1min":0.47199999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83699999999999997,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.442,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76700000000000002,"5min":1.206,"15min":1.206},"last":0.45900000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65100000000000002,"15min":0.67000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.84999999999999998,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.47699999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80200000000000005,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.47699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28400,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343440,"statfs":{"total":96636764160,"available":96607682560,"internally_reserved":0,"allocated":1417216,"data_stored":941635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.32300000000000001,"15min":0.32200000000000001},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58399999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.374,"15min":0.38100000000000001},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.49399999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.439}]},{"osd":3,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.28499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53700000000000003,"5min":0.626,"15min":0.626},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.38900000000000001,"15min":0.39000000000000001},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.51300000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.74299999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.41799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.753,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59899999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77500000000000002,"5min":0.877,"15min":0.877},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.57899999999999996,"15min":0.56599999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.72299999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.443,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79600000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.45300000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70799999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.73399999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.44500000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78400000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.45300000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.75800000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.67900000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34699999999999998,"15min":0.35399999999999998},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.52000000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36799999999999999,"15min":0.376},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.37},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40500000000000003,"15min":0.41499999999999998},"min":{"1min":0.29099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38600000000000001,"15min":0.39000000000000001},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53600000000000003,"5min":0.629,"15min":0.629},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65000000000000002,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.439}]},{"osd":4,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72899999999999998,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.624},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.44400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77600000000000002,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.48099999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.67800000000000005,"15min":0.71799999999999997},"min":{"1min":0.45800000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.76400000000000001,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.495},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.69299999999999995,"15min":0.73299999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85399999999999998,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":0.56699999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.68300000000000005,"15min":0.70899999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79200000000000004,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.70499999999999996,"15min":0.73399999999999999},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":2.149,"15min":2.149},"last":0.58299999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65100000000000002,"15min":0.66600000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81599999999999995,"5min":1.246,"15min":1.246},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.59099999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73499999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32400000000000001,"15min":0.33200000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.497},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34000000000000002,"15min":0.34599999999999997},"min":{"1min":0.23300000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.499,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.56200000000000006}]},{"osd":1,"last update":"Fri Nov 24 06:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33800000000000002,"15min":0.34200000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.25600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.58199999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.47499999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.28599999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45500000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.40899999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.753,"5min":0.755,"15min":0.755},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85999999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.72299999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.753,"5min":0.874,"15min":0.874},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.622,"15min":0.623},"min":{"1min":0.441,"5min":0.433,"15min":0.433},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.73799999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.622},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.623,"15min":0.62},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83099999999999996,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.70099999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61199999999999999,"15min":0.60799999999999998},"min":{"1min":0.48899999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.66200000000000003,"15min":0.67000000000000004},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92100000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.64400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.64200000000000002,"15min":0.68600000000000005},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68799999999999994,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.503},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54400000000000004,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69399999999999995,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60399999999999998,"15min":0.60699999999999998},"min":{"1min":0.42399999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.77000000000000002,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.59999999999999998,"15min":0.59099999999999997},"min":{"1min":0.53700000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55100000000000005}]},{"osd":2,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.436,"5min":0.374,"15min":0.374},"max":{"1min":0.753,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69699999999999995,"5min":0.876,"15min":0.876},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39500000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71399999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.623},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83599999999999997,"5min":0.879,"15min":0.879},"last":0.53500000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53500000000000003,"15min":0.54000000000000004},"min":{"1min":0.33300000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55100000000000005,"15min":0.56200000000000006},"min":{"1min":0.36299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.36299999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55000000000000004,"15min":0.55700000000000005},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.622,"15min":0.628},"min":{"1min":0.42699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:21:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61399999999999999,"15min":0.627},"min":{"1min":0.438,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.90300000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.438},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.67200000000000004,"15min":0.68799999999999994},"min":{"1min":0.47099999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47399999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":351267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57999999999999996,"15min":0.59999999999999998},"min":{"1min":0.38700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78800000000000003,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72699999999999998,"5min":3.04,"15min":3.04},"last":0.63300000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.82799999999999996,"15min":0.95299999999999996},"min":{"1min":0.37,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.93999999999999995,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.435,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73899999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.42499999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71499999999999997,"5min":1.087,"15min":1.087},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.85199999999999998,"15min":0.97999999999999998},"min":{"1min":0.38300000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.99399999999999999,"5min":9.1319999999999997,"15min":9.1319999999999997},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.89600000000000002,"15min":1.0369999999999999},"min":{"1min":0.46600000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.76700000000000002,"5min":10.198,"15min":10.198},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.91100000000000003,"15min":1.0589999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82799999999999996,"5min":10.215999999999999,"15min":10.215999999999999},"last":0.496}]},{"osd":4,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58199999999999996,"15min":0.58299999999999996},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69699999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.67600000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71599999999999997,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.65200000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:21:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.754,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.70899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:21:52.598 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-24T06:21:52.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.714+0000 7f3c7d101700 1 Processor -- start 2023-11-24T06:21:52.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.714+0000 7f3c7d101700 1 -- start start 2023-11-24T06:21:52.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7d101700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3c7815e080 0x7f3c7806bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7d101700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3c780ce430 0x7f3c780ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7d101700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c780cedc0 con 0x7f3c780b4310 2023-11-24T06:21:52.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7d101700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780d1200 con 0x7f3c780ce430 2023-11-24T06:21:52.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7d101700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7815e460 con 0x7f3c7815e080 2023-11-24T06:21:52.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7759e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3c780ce430 0x7f3c780ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7759e700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3c780ce430 0x7f3c780ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39970/0 (socket says 172.21.15.89:39970) 2023-11-24T06:21:52.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7759e700 1 -- 172.21.15.89:0/2337744558 learned_addr learned my addr 172.21.15.89:0/2337744558 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:52.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c76d9d700 1 --2- 172.21.15.89:0/2337744558 >> v2:172.21.15.123:3300/0 conn(0x7f3c7815e080 0x7f3c7806bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7759e700 1 -- 172.21.15.89:0/2337744558 >> v2:172.21.15.123:3300/0 conn(0x7f3c7815e080 msgr2=0x7f3c7806bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:52.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7759e700 1 --2- 172.21.15.89:0/2337744558 >> v2:172.21.15.123:3300/0 conn(0x7f3c7815e080 0x7f3c7806bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7759e700 1 -- 172.21.15.89:0/2337744558 >> v1:172.21.15.123:6789/0 conn(0x7f3c780b4310 legacy=0x7f3c780d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:52.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7759e700 1 -- 172.21.15.89:0/2337744558 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7806ba50 con 0x7f3c780ce430 2023-11-24T06:21:52.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c76d9d700 1 --2- 172.21.15.89:0/2337744558 >> v2:172.21.15.123:3300/0 conn(0x7f3c7815e080 0x7f3c7806bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:21:52.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c7759e700 1 --2- 172.21.15.89:0/2337744558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780ce430 0x7f3c780ce880 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3c780715e0 tx=0x7f3c7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d0cd12ff4d5b03 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:52.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c75d9b700 1 -- 172.21.15.89:0/2337744558 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000e070 con 0x7f3c780ce430 2023-11-24T06:21:52.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c75d9b700 1 -- 172.21.15.89:0/2337744558 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70003a20 con 0x7f3c780ce430 2023-11-24T06:21:52.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.715+0000 7f3c75d9b700 1 -- 172.21.15.89:0/2337744558 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70003dd0 con 0x7f3c780ce430 2023-11-24T06:21:52.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.716+0000 7f3c7d101700 1 -- 172.21.15.89:0/2337744558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780ce430 msgr2=0x7f3c780ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.716+0000 7f3c7d101700 1 --2- 172.21.15.89:0/2337744558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780ce430 0x7f3c780ce880 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3c780715e0 tx=0x7f3c7000b330 comp rx=0 tx=0).stop 2023-11-24T06:21:52.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.716+0000 7f3c7d101700 1 -- 172.21.15.89:0/2337744558 shutdown_connections 2023-11-24T06:21:52.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.716+0000 7f3c7d101700 1 --2- 172.21.15.89:0/2337744558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780ce430 0x7f3c780ce880 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.716+0000 7f3c7d101700 1 --2- 172.21.15.89:0/2337744558 >> v2:172.21.15.123:3300/0 conn(0x7f3c7815e080 0x7f3c7806bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.716+0000 7f3c7d101700 1 -- 172.21.15.89:0/2337744558 >> 172.21.15.89:0/2337744558 conn(0x7f3c780b3780 msgr2=0x7f3c780b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:52.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.716+0000 7f3c7d101700 1 -- 172.21.15.89:0/2337744558 shutdown_connections 2023-11-24T06:21:52.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.716+0000 7f3c7d101700 1 -- 172.21.15.89:0/2337744558 wait complete. 2023-11-24T06:21:52.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.717+0000 7f3c7d101700 1 Processor -- start 2023-11-24T06:21:52.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c7d101700 1 -- start start 2023-11-24T06:21:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c7d101700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780b4310 0x7f3c78155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c7d101700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3c780ce430 0x7f3c780c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c7d101700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c7815e080 0x7f3c780b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c7d101700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780b50c0 con 0x7f3c780b4310 2023-11-24T06:21:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c7d101700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780b53a0 con 0x7f3c7815e080 2023-11-24T06:21:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c7d101700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780b5680 con 0x7f3c780ce430 2023-11-24T06:21:52.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c76d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780b4310 0x7f3c78155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c76d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780b4310 0x7f3c78155c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39972/0 (socket says 172.21.15.89:39972) 2023-11-24T06:21:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c76d9d700 1 -- 172.21.15.89:0/957917081 learned_addr learned my addr 172.21.15.89:0/957917081 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.718+0000 7f3c7759e700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c7815e080 0x7f3c780b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c7659c700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3c780ce430 0x7f3c780c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c76d9d700 1 -- 172.21.15.89:0/957917081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3c780ce430 msgr2=0x7f3c780c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c76d9d700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3c780ce430 0x7f3c780c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c76d9d700 1 -- 172.21.15.89:0/957917081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c7815e080 msgr2=0x7f3c780b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c76d9d700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c7815e080 0x7f3c780b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c76d9d700 1 -- 172.21.15.89:0/957917081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c70009b30 con 0x7f3c780b4310 2023-11-24T06:21:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c76d9d700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780b4310 0x7f3c78155c40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800c7f0 tx=0x7f3c6800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c5f7fe700 1 -- 172.21.15.89:0/957917081 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68011800 con 0x7f3c780b4310 2023-11-24T06:21:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c5f7fe700 1 -- 172.21.15.89:0/957917081 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c68011960 con 0x7f3c780b4310 2023-11-24T06:21:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c5f7fe700 1 -- 172.21.15.89:0/957917081 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68011cf0 con 0x7f3c780b4310 2023-11-24T06:21:52.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c78156340 con 0x7f3c780b4310 2023-11-24T06:21:52.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.719+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c780b1e40 con 0x7f3c780b4310 2023-11-24T06:21:52.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.720+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c781561a0 con 0x7f3c780b4310 2023-11-24T06:21:52.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.720+0000 7f3c5f7fe700 1 -- 172.21.15.89:0/957917081 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f3c68018360 con 0x7f3c780b4310 2023-11-24T06:21:52.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.721+0000 7f3c5f7fe700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3c6406aff0 0x7f3c6406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:52.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.721+0000 7f3c5f7fe700 1 -- 172.21.15.89:0/957917081 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3c68089380 con 0x7f3c780b4310 2023-11-24T06:21:52.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.721+0000 7f3c7659c700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3c6406aff0 0x7f3c6406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:52.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.721+0000 7f3c7659c700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3c6406aff0 0x7f3c6406d4b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3c7806ea30 tx=0x7f3c6c007ec0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:52.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.725+0000 7f3c5f7fe700 1 -- 172.21.15.89:0/957917081 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c781561a0 con 0x7f3c780b4310 2023-11-24T06:21:52.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.914+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f3c781561a0 con 0x7f3c780b4310 2023-11-24T06:21:52.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.917+0000 7f3c5f7fe700 1 -- 172.21.15.89:0/957917081 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v16)=0 v16) v1 ==== 123+0+0 (secure 0 0 0) 0x7f3c781561a0 con 0x7f3c780b4310 2023-11-24T06:21:52.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.918+0000 7f3c5f7fe700 1 -- 172.21.15.89:0/957917081 <== mon.0 v2:172.21.15.89:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3c68054510 con 0x7f3c780b4310 2023-11-24T06:21:52.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3c6406aff0 msgr2=0x7f3c6406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3c6406aff0 0x7f3c6406d4b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3c7806ea30 tx=0x7f3c6c007ec0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780b4310 msgr2=0x7f3c78155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:52.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780b4310 0x7f3c78155c40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800c7f0 tx=0x7f3c6800cb00 comp rx=0 tx=0).stop 2023-11-24T06:21:52.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 shutdown_connections 2023-11-24T06:21:52.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f3c6406aff0 0x7f3c6406d4b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c7815e080 0x7f3c780b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c780b4310 0x7f3c78155c40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 --2- 172.21.15.89:0/957917081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3c780ce430 0x7f3c780c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:52.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 >> 172.21.15.89:0/957917081 conn(0x7f3c780b3780 msgr2=0x7f3c78070670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:52.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 shutdown_connections 2023-11-24T06:21:52.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:52.920+0000 7f3c7d101700 1 -- 172.21.15.89:0/957917081 wait complete. 2023-11-24T06:21:52.935 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T06:21:52.936 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:21:53.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.053+0000 7f51674e4700 1 Processor -- start 2023-11-24T06:21:53.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.053+0000 7f51674e4700 1 -- start start 2023-11-24T06:21:53.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.054+0000 7f51674e4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600b0fb0 0x7f51600cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:53.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.054+0000 7f51674e4700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51600d0280 0x7f516015d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:53.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.054+0000 7f51674e4700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51600b1c60 con 0x7f51600b1390 2023-11-24T06:21:53.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.054+0000 7f51674e4700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51600d0720 con 0x7f51600d0280 2023-11-24T06:21:53.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.054+0000 7f51674e4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51600d0a00 con 0x7f51600b0fb0 2023-11-24T06:21:53.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.054+0000 7f5165a81700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51600d0280 0x7f516015d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:53.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.054+0000 7f5165a81700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51600d0280 0x7f516015d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39984/0 (socket says 172.21.15.89:39984) 2023-11-24T06:21:53.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.054+0000 7f5165a81700 1 -- 172.21.15.89:0/3921812833 learned_addr learned my addr 172.21.15.89:0/3921812833 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:53.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f5165a81700 1 -- 172.21.15.89:0/3921812833 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600b0fb0 msgr2=0x7f51600cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:53.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f5165a81700 1 --2- 172.21.15.89:0/3921812833 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600b0fb0 0x7f51600cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f5165a81700 1 -- 172.21.15.89:0/3921812833 >> v1:172.21.15.123:6789/0 conn(0x7f51600b1390 legacy=0x7f516006b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:21:53.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f5165a81700 1 -- 172.21.15.89:0/3921812833 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51600d0ce0 con 0x7f51600d0280 2023-11-24T06:21:53.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f5165a81700 1 --2- 172.21.15.89:0/3921812833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600d0280 0x7f516015d920 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5158009e50 tx=0x7f515800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6407c112e2399b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f5153fff700 1 -- 172.21.15.89:0/3921812833 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515800e070 con 0x7f51600d0280 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f5153fff700 1 -- 172.21.15.89:0/3921812833 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5158003a20 con 0x7f51600d0280 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f5153fff700 1 -- 172.21.15.89:0/3921812833 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5158003df0 con 0x7f51600d0280 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f51674e4700 1 -- 172.21.15.89:0/3921812833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600d0280 msgr2=0x7f516015d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.055+0000 7f51674e4700 1 --2- 172.21.15.89:0/3921812833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600d0280 0x7f516015d920 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5158009e50 tx=0x7f515800b330 comp rx=0 tx=0).stop 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.056+0000 7f51674e4700 1 -- 172.21.15.89:0/3921812833 shutdown_connections 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.056+0000 7f51674e4700 1 --2- 172.21.15.89:0/3921812833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600d0280 0x7f516015d920 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.056+0000 7f51674e4700 1 --2- 172.21.15.89:0/3921812833 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600b0fb0 0x7f51600cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.056+0000 7f51674e4700 1 -- 172.21.15.89:0/3921812833 >> 172.21.15.89:0/3921812833 conn(0x7f516009aa00 msgr2=0x7f516009ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:53.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.056+0000 7f51674e4700 1 -- 172.21.15.89:0/3921812833 shutdown_connections 2023-11-24T06:21:53.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.056+0000 7f51674e4700 1 -- 172.21.15.89:0/3921812833 wait complete. 2023-11-24T06:21:53.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.057+0000 7f51674e4700 1 Processor -- start 2023-11-24T06:21:53.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.057+0000 7f51674e4700 1 -- start start 2023-11-24T06:21:53.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.057+0000 7f51674e4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51600b0fb0 0x7f516015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:53.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.057+0000 7f51674e4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600b1390 0x7f51600c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f51674e4700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600d0280 0x7f51600b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f51674e4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516015b770 con 0x7f51600b1390 2023-11-24T06:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f51674e4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51600ca6f0 con 0x7f51600b0fb0 2023-11-24T06:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f51674e4700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51600ca9d0 con 0x7f51600d0280 2023-11-24T06:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600b1390 0x7f51600c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600b1390 0x7f51600c9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39988/0 (socket says 172.21.15.89:39988) 2023-11-24T06:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 -- 172.21.15.89:0/4248460928 learned_addr learned my addr 172.21.15.89:0/4248460928 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-11-24T06:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5165280700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51600b0fb0 0x7f516015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5165a81700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600d0280 0x7f51600b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 -- 172.21.15.89:0/4248460928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600d0280 msgr2=0x7f51600b5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600d0280 0x7f51600b5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 -- 172.21.15.89:0/4248460928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51600b0fb0 msgr2=0x7f516015b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51600b0fb0 0x7f516015b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 -- 172.21.15.89:0/4248460928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5158009b30 con 0x7f51600b1390 2023-11-24T06:21:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5164a7f700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600b1390 0x7f51600c9c50 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f515c00c4a0 tx=0x7f515c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:53.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5151ffb700 1 -- 172.21.15.89:0/4248460928 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c0054e0 con 0x7f51600b1390 2023-11-24T06:21:53.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5151ffb700 1 -- 172.21.15.89:0/4248460928 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f515c003680 con 0x7f51600b1390 2023-11-24T06:21:53.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51600cacb0 con 0x7f51600b1390 2023-11-24T06:21:53.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f5151ffb700 1 -- 172.21.15.89:0/4248460928 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c003a50 con 0x7f51600b1390 2023-11-24T06:21:53.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.058+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51600cb270 con 0x7f51600b1390 2023-11-24T06:21:53.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.059+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51600c5ed0 con 0x7f51600b1390 2023-11-24T06:21:53.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.060+0000 7f5151ffb700 1 -- 172.21.15.89:0/4248460928 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 89333+0+0 (secure 0 0 0) 0x7f515c003bb0 con 0x7f51600b1390 2023-11-24T06:21:53.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.060+0000 7f5151ffb700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f514406aff0 0x7f514406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:21:53.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.060+0000 7f5151ffb700 1 -- 172.21.15.89:0/4248460928 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f515c088ad0 con 0x7f51600b1390 2023-11-24T06:21:53.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.060+0000 7f5165280700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f514406aff0 0x7f514406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:21:53.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.061+0000 7f5165280700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f514406aff0 0x7f514406d4b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5154006fd0 tx=0x7f5154008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:21:53.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.065+0000 7f5151ffb700 1 -- 172.21.15.89:0/4248460928 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f515c053c50 con 0x7f51600b1390 2023-11-24T06:21:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.297+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5160068480 con 0x7f51600b1390 2023-11-24T06:21:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.297+0000 7f5151ffb700 1 -- 172.21.15.89:0/4248460928 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f515c017070 con 0x7f51600b1390 2023-11-24T06:21:53.301 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-11-24T06:21:53.301 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5387,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3825764763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":3825764763},{"type":"v1","addr":"172.21.15.123:6835","nonce":3825764763}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/351443905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":351443905},{"type":"v1","addr":"172.21.15.123:6834","nonce":351443905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6021,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2890850053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2890850053},{"type":"v1","addr":"172.21.15.89:6836","nonce":2890850053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3912248946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3912248946},{"type":"v1","addr":"172.21.15.89:6837","nonce":3912248946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[]} 2023-11-24T06:21:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f514406aff0 msgr2=0x7f514406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f514406aff0 0x7f514406d4b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5154006fd0 tx=0x7f5154008040 comp rx=0 tx=0).stop 2023-11-24T06:21:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600b1390 msgr2=0x7f51600c9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:21:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600b1390 0x7f51600c9c50 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f515c00c4a0 tx=0x7f515c00c860 comp rx=0 tx=0).stop 2023-11-24T06:21:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 shutdown_connections 2023-11-24T06:21:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600d0280 0x7f51600b5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:6832/2291342889,v1:172.21.15.89:6833/2291342889] conn(0x7f514406aff0 0x7f514406d4b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51600b0fb0 0x7f516015b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 --2- 172.21.15.89:0/4248460928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51600b1390 0x7f51600c9c50 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:21:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 >> 172.21.15.89:0/4248460928 conn(0x7f516009aa00 msgr2=0x7f51600701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:21:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 shutdown_connections 2023-11-24T06:21:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-11-24T06:21:53.300+0000 7f51674e4700 1 -- 172.21.15.89:0/4248460928 wait complete. 2023-11-24T06:21:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 44 2023-11-24T06:21:53.315 DEBUG:tasks.cephfs.filesystem:already dead... 2023-11-24T06:21:53.315 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-24T06:21:53.315 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T06:21:53.315 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.324 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T06:21:53.325 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T06:21:53.325 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.374 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T06:21:53.374 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T06:21:53.375 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.387 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T06:21:53.387 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T06:21:53.387 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.439 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T06:21:53.439 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-24T06:21:53.440 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-24T06:21:53.440 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.475 INFO:tasks.ceph.osd.0:Stopped 2023-11-24T06:21:53.476 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-24T06:21:53.476 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.541 INFO:tasks.ceph.osd.1:Stopped 2023-11-24T06:21:53.542 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-24T06:21:53.542 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.617 INFO:tasks.ceph.osd.2:Stopped 2023-11-24T06:21:53.617 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-24T06:21:53.617 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.699 INFO:tasks.ceph.osd.3:Stopped 2023-11-24T06:21:53.699 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-24T06:21:53.699 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.741 INFO:tasks.ceph.osd.4:Stopped 2023-11-24T06:21:53.741 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-24T06:21:53.741 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.776 INFO:tasks.ceph.osd.5:Stopped 2023-11-24T06:21:53.776 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-24T06:21:53.776 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.844 INFO:tasks.ceph.osd.6:Stopped 2023-11-24T06:21:53.845 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-24T06:21:53.845 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.917 INFO:tasks.ceph.osd.7:Stopped 2023-11-24T06:21:53.917 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-24T06:21:53.917 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-24T06:21:53.918 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.942 INFO:tasks.ceph.mgr.y:Stopped 2023-11-24T06:21:53.942 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-24T06:21:53.943 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.983 INFO:tasks.ceph.mgr.x:Stopped 2023-11-24T06:21:53.983 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-24T06:21:53.983 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-24T06:21:53.983 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:53.995 INFO:tasks.ceph.mon.a:Stopped 2023-11-24T06:21:53.995 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-24T06:21:53.995 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:54.036 INFO:tasks.ceph.mon.b:Stopped 2023-11-24T06:21:54.036 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-24T06:21:54.037 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:21:54.088 INFO:tasks.ceph.mon.c:Stopped 2023-11-24T06:21:54.089 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-24T06:21:54.089 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2023-11-24T06:21:54.123 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:21:54.123 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-24T06:21:54.351 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:21:54.351 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-24T06:21:54.451 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:21:54.452 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-24T06:21:54.535 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:21:54.535 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-24T06:21:54.600 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:21:54.600 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-24T06:21:54.843 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:21:54.843 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-24T06:21:54.933 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:21:54.934 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-24T06:21:55.005 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:21:55.005 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-24T06:21:55.100 INFO:tasks.ceph:Archiving mon data... 2023-11-24T06:21:55.101 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/data/mon.a.tgz 2023-11-24T06:21:55.101 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2023-11-24T06:21:55.119 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.owbd6bDxJ2 2023-11-24T06:21:55.119 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.owbd6bDxJ2 2023-11-24T06:21:55.271 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.owbd6bDxJ2 2023-11-24T06:21:55.357 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.owbd6bDxJ2 is 441KB 2023-11-24T06:21:55.454 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.owbd6bDxJ2 2023-11-24T06:21:55.471 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/data/mon.b.tgz 2023-11-24T06:21:55.471 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-11-24T06:21:55.492 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.5PkiOp5Za1 2023-11-24T06:21:55.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.5PkiOp5Za1 2023-11-24T06:21:55.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.5PkiOp5Za1 2023-11-24T06:21:55.762 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.5PkiOp5Za1 is 586KB 2023-11-24T06:21:55.860 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.5PkiOp5Za1 2023-11-24T06:21:55.876 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/data/mon.c.tgz 2023-11-24T06:21:55.876 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-11-24T06:21:55.932 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.sa74n4hn40 2023-11-24T06:21:55.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.sa74n4hn40 2023-11-24T06:21:56.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.sa74n4hn40 2023-11-24T06:21:56.185 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.sa74n4hn40 is 586KB 2023-11-24T06:21:56.286 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.sa74n4hn40 2023-11-24T06:21:56.303 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-24T06:21:56.303 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-24T06:21:56.306 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-24T06:21:56.368 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-24T06:21:56.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-24T06:21:56.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-24T06:21:56.413 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-24T06:21:56.444 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-24T06:21:56.449 INFO:tasks.ceph:Archiving crash dumps... 2023-11-24T06:21:56.452 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/remote/smithi089/crash 2023-11-24T06:21:56.452 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-24T06:21:56.660 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/remote/smithi123/crash 2023-11-24T06:21:56.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-24T06:21:56.734 INFO:tasks.ceph:Shutting down logrotate 2023-11-24T06:21:56.735 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:21:56.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:21:56.841 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-24T06:21:56.871 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-24T06:21:56.898 INFO:tasks.ceph:Compressing logs... 2023-11-24T06:21:56.899 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-24T06:21:56.915 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-24T06:21:56.941 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110164.log 2023-11-24T06:21:56.942 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122536.log 2023-11-24T06:21:56.942 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115102.log 2023-11-24T06:21:56.943 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120255.log 2023-11-24T06:21:56.943 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110164.log.gz 2023-11-24T06:21:56.943 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3%/var/log/ceph/ceph-client.admin.115102.log: -- replaced with /var/log/ceph/ceph-client.admin.122536.log.gz 2023-11-24T06:21:56.944 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110302.log 2023-11-24T06:21:56.944 INFO:teuthology.orchestra.run.smithi089.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.115102.log.gz 2023-11-24T06:21:56.944 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123242.log 2023-11-24T06:21:56.985 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-24T06:21:56.986 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.112197.log 2023-11-24T06:21:56.986 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103537.log 2023-11-24T06:21:56.986 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.1.112197.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-24T06:21:56.986 INFO:teuthology.orchestra.run.smithi123.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.112197.log.gz 2023-11-24T06:21:56.986 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.103537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.110173.log 2023-11-24T06:21:56.986 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103537.log.gz 2023-11-24T06:21:56.987 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103492.log 2023-11-24T06:21:56.987 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103627.log 2023-11-24T06:21:56.987 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.1.110173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.110173.log.gz 2023-11-24T06:21:56.987 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.103492.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-24T06:21:56.987 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103492.log.gz 2023-11-24T06:21:56.987 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.103627.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-11-24T06:21:56.988 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-24T06:21:56.988 INFO:teuthology.orchestra.run.smithi123.stderr: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-24T06:21:56.988 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.6.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.103627.log.gz 2023-11-24T06:21:56.988 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103353.log 2023-11-24T06:21:56.988 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-24T06:21:56.988 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.103353.log: /var/log/ceph/ceph-osd.4.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103353.log.gz 2023-11-24T06:21:56.988 INFO:teuthology.orchestra.run.smithi123.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-24T06:21:56.989 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.110788.log 2023-11-24T06:21:56.989 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.112802.log 2023-11-24T06:21:56.989 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.3.110788.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-24T06:21:56.989 INFO:teuthology.orchestra.run.smithi123.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.3.110788.log.gz 2023-11-24T06:21:56.989 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.1.112802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.110792.log 2023-11-24T06:21:56.989 INFO:teuthology.orchestra.run.smithi123.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.112802.log.gz 2023-11-24T06:21:56.989 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.3.110792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.110792.log.gz 2023-11-24T06:21:56.989 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.110181.log 2023-11-24T06:21:56.990 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.112806.log 2023-11-24T06:21:56.990 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.1.110181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.110181.log.gz 2023-11-24T06:21:56.990 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.1.112806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.112806.log.gz 2023-11-24T06:21:56.990 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103680.log 2023-11-24T06:21:56.990 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-24T06:21:56.990 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.103680.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103680.log.gz 2023-11-24T06:21:56.990 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-24T06:21:56.991 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.111587.log 2023-11-24T06:21:56.991 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.112201.log 2023-11-24T06:21:56.991 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.1.111587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.111587.log.gz 2023-11-24T06:21:56.991 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.1.112201.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.103400.log 2023-11-24T06:21:56.991 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.1.112201.log.gz 2023-11-24T06:21:56.991 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-24T06:21:56.991 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.103400.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.103400.log.gz 2023-11-24T06:21:56.991 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103445.log 2023-11-24T06:21:56.992 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.103445.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103445.log.gz 2023-11-24T06:21:56.992 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-24T06:21:56.992 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.111591.log 2023-11-24T06:21:56.993 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.1.111591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.111591.log.gz 2023-11-24T06:21:56.994 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-11-24T06:21:56.994 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103582.log 2023-11-24T06:21:57.010 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120255.log: /var/log/ceph/ceph-client.admin.110302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114263.log 2023-11-24T06:21:57.010 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.120255.log.gz 2023-11-24T06:21:57.010 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123242.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110302.log.gz 2023-11-24T06:21:57.010 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117259.log 2023-11-24T06:21:57.010 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123242.log.gz 2023-11-24T06:21:57.010 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115558.log 2023-11-24T06:21:57.011 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114263.log: /var/log/ceph/ceph-client.admin.117259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119979.log 2023-11-24T06:21:57.011 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114263.log.gz 2023-11-24T06:21:57.011 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5%/var/log/ceph/ceph-client.admin.115558.log: -- replaced with /var/log/ceph/ceph-client.admin.117259.log.gz 2023-11-24T06:21:57.011 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114309.log 2023-11-24T06:21:57.011 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125050.log 2023-11-24T06:21:57.011 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.115558.log.gz 2023-11-24T06:21:57.011 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114309.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.119979.log.gz 2023-11-24T06:21:57.012 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125096.log 2023-11-24T06:21:57.012 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114309.log.gz 2023-11-24T06:21:57.012 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122045.log 2023-11-24T06:21:57.012 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111100.log 2023-11-24T06:21:57.012 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.125050.log.gz 2023-11-24T06:21:57.012 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126225.log 2023-11-24T06:21:57.012 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122045.log.gz 2023-11-24T06:21:57.012 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128354.log 2023-11-24T06:21:57.013 INFO:teuthology.orchestra.run.smithi089.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.125096.log.gz 2023-11-24T06:21:57.013 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.111100.log.gz 2023-11-24T06:21:57.013 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112096.log 2023-11-24T06:21:57.013 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126225.log: /var/log/ceph/ceph-client.admin.128354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121632.log 2023-11-24T06:21:57.013 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126225.log.gz 2023-11-24T06:21:57.013 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128354.log.gz 2023-11-24T06:21:57.013 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116163.log 2023-11-24T06:21:57.013 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.112096.log.gz 2023-11-24T06:21:57.014 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126179.log 2023-11-24T06:21:57.014 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110118.log 2023-11-24T06:21:57.014 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126179.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116163.log.gz 2023-11-24T06:21:57.014 INFO:teuthology.orchestra.run.smithi089.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.121632.log.gz 2023-11-24T06:21:57.014 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110732.log 2023-11-24T06:21:57.014 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126179.log.gz 2023-11-24T06:21:57.014 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119795.log 2023-11-24T06:21:57.015 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110118.log.gz 2023-11-24T06:21:57.015 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112050.log 2023-11-24T06:21:57.015 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110732.log.gz 2023-11-24T06:21:57.015 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110532.log 2023-11-24T06:21:57.015 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112050.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119795.log.gz 2023-11-24T06:21:57.015 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114494.log 2023-11-24T06:21:57.015 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112050.log.gz 2023-11-24T06:21:57.015 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123288.log 2023-11-24T06:21:57.016 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110532.log.gz 2023-11-24T06:21:57.016 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116485.log 2023-11-24T06:21:57.016 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123288.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114494.log.gz 2023-11-24T06:21:57.016 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109809.log 2023-11-24T06:21:57.016 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123288.log.gz 2023-11-24T06:21:57.016 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120836.log 2023-11-24T06:21:57.016 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116485.log.gz 2023-11-24T06:21:57.016 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127047.log 2023-11-24T06:21:57.017 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109809.log: /var/log/ceph/ceph-client.admin.120836.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-24T06:21:57.017 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109809.log.gz 2023-11-24T06:21:57.017 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127406.log 2023-11-24T06:21:57.017 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.120836.log.gz 2023-11-24T06:21:57.017 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.2.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.127047.log.gz-5 2023-11-24T06:21:57.017 INFO:teuthology.orchestra.run.smithi089.stderr:--verbose -- /var/log/ceph/ceph-client.admin.121402.log 2023-11-24T06:21:57.017 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116531.log 2023-11-24T06:21:57.017 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.127406.log.gz 2023-11-24T06:21:57.018 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125647.log 2023-11-24T06:21:57.018 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124544.log 2023-11-24T06:21:57.018 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.121402.log.gz 2023-11-24T06:21:57.018 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125647.log: gzip -5 --verbose -- 89.3%/var/log/ceph/ceph-client.admin.116761.log 2023-11-24T06:21:57.018 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/ceph-client.admin.116531.log.gz 2023-11-24T06:21:57.018 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.125647.log.gz 2023-11-24T06:21:57.018 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116025.log 2023-11-24T06:21:57.019 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124544.log: /var/log/ceph/ceph-client.admin.116761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109901.log 2023-11-24T06:21:57.019 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124544.log.gz 2023-11-24T06:21:57.019 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116761.log.gz 2023-11-24T06:21:57.019 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113941.log 2023-11-24T06:21:57.019 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115420.log 2023-11-24T06:21:57.019 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116025.log.gz 2023-11-24T06:21:57.019 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.113941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119224.log 2023-11-24T06:21:57.019 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109901.log.gz 2023-11-24T06:21:57.020 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113941.log.gz 2023-11-24T06:21:57.020 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116393.log 2023-11-24T06:21:57.020 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120117.log 2023-11-24T06:21:57.020 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115420.log.gz 2023-11-24T06:21:57.020 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116393.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119224.log.gz 2023-11-24T06:21:57.020 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-24T06:21:57.020 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120117.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116393.log.gz 2023-11-24T06:21:57.020 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117836.log 2023-11-24T06:21:57.021 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120117.log.gz 2023-11-24T06:21:57.021 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120545.log 2023-11-24T06:21:57.021 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123696.log 2023-11-24T06:21:57.021 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4%/var/log/ceph/ceph-client.admin.120545.log: -- replaced with /var/log/ceph/ceph-client.admin.117836.log.gz 2023-11-24T06:21:57.021 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122809.log 2023-11-24T06:21:57.021 INFO:teuthology.orchestra.run.smithi089.stderr: 85.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-24T06:21:57.021 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.120545.log.gz 2023-11-24T06:21:57.022 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124682.log 2023-11-24T06:21:57.022 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122809.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123696.log.gz 2023-11-24T06:21:57.022 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115979.log 2023-11-24T06:21:57.022 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.122809.log.gz 2023-11-24T06:21:57.022 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121126.log 2023-11-24T06:21:57.022 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124682.log: /var/log/ceph/ceph-client.admin.115979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121264.log 2023-11-24T06:21:57.022 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.124682.log.gz 2023-11-24T06:21:57.025 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115979.log.gz 2023-11-24T06:21:57.025 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126271.log 2023-11-24T06:21:57.025 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126725.log 2023-11-24T06:21:57.025 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121264.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121126.log.gz 2023-11-24T06:21:57.025 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111475.log 2023-11-24T06:21:57.025 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.121264.log.gz 2023-11-24T06:21:57.026 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126271.log.gz 2023-11-24T06:21:57.026 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123950.log 2023-11-24T06:21:57.026 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110394.log 2023-11-24T06:21:57.026 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.126725.log.gz 2023-11-24T06:21:57.026 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123950.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111475.log.gz 2023-11-24T06:21:57.026 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127498.log 2023-11-24T06:21:57.026 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110394.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.123950.log.gz 2023-11-24T06:21:57.026 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123444.log 2023-11-24T06:21:57.027 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110394.log.gz 2023-11-24T06:21:57.027 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120163.log 2023-11-24T06:21:57.027 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.127498.log.gz 2023-11-24T06:21:57.027 INFO:teuthology.orchestra.run.smithi089.stderr:gzip /var/log/ceph/ceph-client.admin.123444.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.96928.log 2023-11-24T06:21:57.027 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125861.log 2023-11-24T06:21:57.027 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.123444.log.gz 2023-11-24T06:21:57.027 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.tmp-client.admin.96928.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.120163.log.gz 2023-11-24T06:21:57.028 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 0.0%-5 --verbose -- -- replaced with /var/log/ceph/ceph.tmp-client.admin.96928.log.gz/var/log/ceph/ceph-client.admin.111469.log 2023-11-24T06:21:57.028 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-11-24T06:21:57.028 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124264.log 2023-11-24T06:21:57.028 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125861.log.gz 2023-11-24T06:21:57.028 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109763.log 2023-11-24T06:21:57.028 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111469.log.gz 2023-11-24T06:21:57.028 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124171.log 2023-11-24T06:21:57.028 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124264.log: /var/log/ceph/ceph-client.admin.109763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116117.log 2023-11-24T06:21:57.029 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.124264.log.gz 2023-11-24T06:21:57.029 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109763.log.gz 2023-11-24T06:21:57.029 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111472.log 2023-11-24T06:21:57.029 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124171.log.gz 2023-11-24T06:21:57.029 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117397.log 2023-11-24T06:21:57.029 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116117.log: /var/log/ceph/ceph-client.admin.111472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127636.log 2023-11-24T06:21:57.029 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116117.log.gz 2023-11-24T06:21:57.029 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117397.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111472.log.gz 2023-11-24T06:21:57.030 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116209.log 2023-11-24T06:21:57.030 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.117397.log.gz 2023-11-24T06:21:57.030 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126863.log 2023-11-24T06:21:57.030 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.127636.log.gz 2023-11-24T06:21:57.030 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115374.log 2023-11-24T06:21:57.030 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116209.log: /var/log/ceph/ceph-client.admin.126863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122583.log 2023-11-24T06:21:57.030 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116209.log.gz 2023-11-24T06:21:57.031 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115374.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.126863.log.gz 2023-11-24T06:21:57.031 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111103.log 2023-11-24T06:21:57.031 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115374.log.gz 2023-11-24T06:21:57.031 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125693.log 2023-11-24T06:21:57.031 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.122583.log.gz 2023-11-24T06:21:57.031 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120025.log 2023-11-24T06:21:57.031 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125953.log 2023-11-24T06:21:57.031 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111103.log.gz 2023-11-24T06:21:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120025.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.125693.log.gz 2023-11-24T06:21:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126679.log 2023-11-24T06:21:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:gzip /var/log/ceph/ceph-client.admin.125953.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.124358.log 2023-11-24T06:21:57.032 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120025.log.gz 2023-11-24T06:21:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116255.log 2023-11-24T06:21:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126679.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.125953.log.gz 2023-11-24T06:21:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111473.log 2023-11-24T06:21:57.032 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126679.log.gz 2023-11-24T06:21:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116255.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124358.log.gz 2023-11-24T06:21:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-24T06:21:57.033 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116255.log.gzgzip 2023-11-24T06:21:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.110440.log 2023-11-24T06:21:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111473.log: /var/log/ceph/ceph-osd.0.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111473.log.gz 2023-11-24T06:21:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124452.log 2023-11-24T06:21:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112326.log 2023-11-24T06:21:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110440.log: /var/log/ceph/ceph-client.admin.124452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.121935.log 2023-11-24T06:21:57.034 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110440.log.gz 2023-11-24T06:21:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116623.log 2023-11-24T06:21:57.034 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.124452.log.gz 2023-11-24T06:21:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.121935.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112326.log.gz 2023-11-24T06:21:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119749.log 2023-11-24T06:21:57.034 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.121935.log.gz 2023-11-24T06:21:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.113669.log 2023-11-24T06:21:57.035 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119749.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116623.log.gz 2023-11-24T06:21:57.035 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128050.log 2023-11-24T06:21:57.035 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119749.log.gz 2023-11-24T06:21:57.035 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.2.113669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.113669.log.gz 2023-11-24T06:21:57.035 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111314.log 2023-11-24T06:21:57.035 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.128050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111096.log 2023-11-24T06:21:57.035 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111314.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128050.log.gz 2023-11-24T06:21:57.035 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124866.log 2023-11-24T06:21:57.036 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111314.log.gz 2023-11-24T06:21:57.036 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109855.log 2023-11-24T06:21:57.036 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119408.log 2023-11-24T06:21:57.036 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4%/var/log/ceph/ceph-client.admin.124866.log: -- replaced with /var/log/ceph/ceph-client.admin.111096.log.gz 2023-11-24T06:21:57.036 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109855.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124866.log.gz 2023-11-24T06:21:57.036 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-11-24T06:21:57.036 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109855.log.gz 2023-11-24T06:21:57.037 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 /var/log/ceph/ceph-client.admin.119408.log: --verbose -- /var/log/ceph/ceph-client.admin.118729.log 2023-11-24T06:21:57.037 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112188.log 2023-11-24T06:21:57.037 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119408.log.gz 2023-11-24T06:21:57.037 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127314.log 2023-11-24T06:21:57.037 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118729.log.gz 2023-11-24T06:21:57.037 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112188.log: /var/log/ceph/ceph-client.admin.127314.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112188.log.gz 2023-11-24T06:21:57.037 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127314.log.gz 2023-11-24T06:21:57.038 INFO:teuthology.orchestra.run.smithi089.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-11-24T06:21:57.038 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122920.log 2023-11-24T06:21:57.038 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110719.log 2023-11-24T06:21:57.038 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112142.log 2023-11-24T06:21:57.038 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.122920.log.gz 2023-11-24T06:21:57.038 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117631.log 2023-11-24T06:21:57.038 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112142.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110719.log.gz 2023-11-24T06:21:57.038 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123150.log 2023-11-24T06:21:57.039 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117631.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112142.log.gz 2023-11-24T06:21:57.039 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120301.log 2023-11-24T06:21:57.039 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.117631.log.gz 2023-11-24T06:21:57.039 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.113016.log 2023-11-24T06:21:57.039 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114699.log /var/log/ceph/ceph-client.admin.120301.log: 2023-11-24T06:21:57.039 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123150.log.gz 2023-11-24T06:21:57.039 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.113016.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120301.log.gz 2023-11-24T06:21:57.040 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125555.log 2023-11-24T06:21:57.040 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.113016.log.gz 2023-11-24T06:21:57.040 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117933.log 2023-11-24T06:21:57.040 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114699.log.gz 2023-11-24T06:21:57.040 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127866.log 2023-11-24T06:21:57.040 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125555.log.gz 2023-11-24T06:21:57.040 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123796.log 2023-11-24T06:21:57.040 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117933.log: /var/log/ceph/ceph-client.admin.127866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109430.log 2023-11-24T06:21:57.041 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117933.log.gz 2023-11-24T06:21:57.041 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124590.log 2023-11-24T06:21:57.041 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127866.log.gz 2023-11-24T06:21:57.041 INFO:teuthology.orchestra.run.smithi089.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.123796.log.gz 2023-11-24T06:21:57.041 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121586.log 2023-11-24T06:21:57.041 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124590.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109430.log.gz 2023-11-24T06:21:57.041 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-24T06:21:57.042 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124590.log.gz 2023-11-24T06:21:57.042 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126771.log 2023-11-24T06:21:57.042 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.121586.log.gz 2023-11-24T06:21:57.042 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125509.log 2023-11-24T06:21:57.042 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124820.log /var/log/ceph/ceph-client.admin.126771.log: 2023-11-24T06:21:57.042 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125509.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126771.log.gz 2023-11-24T06:21:57.042 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124820.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125509.log.gz 2023-11-24T06:21:57.042 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.124820.log.gz 2023-11-24T06:21:57.043 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117305.log 2023-11-24T06:21:57.043 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121032.log 2023-11-24T06:21:57.043 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114033.log 2023-11-24T06:21:57.043 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117305.log.gz 2023-11-24T06:21:57.043 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121032.log: /var/log/ceph/ceph-client.admin.114033.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121032.log.gz 2023-11-24T06:21:57.043 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114033.log.gz 2023-11-24T06:21:57.043 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119887.log 2023-11-24T06:21:57.044 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109970.log 2023-11-24T06:21:57.044 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125601.log 2023-11-24T06:21:57.044 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109970.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119887.log.gz 2023-11-24T06:21:57.045 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109970.log.gz 2023-11-24T06:21:57.045 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125601.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.125601.log.gz 2023-11-24T06:21:57.045 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122137.log 2023-11-24T06:21:57.045 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124310.log 2023-11-24T06:21:57.045 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127728.log 2023-11-24T06:21:57.045 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124310.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.122137.log.gz 2023-11-24T06:21:57.046 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124310.log.gz 2023-11-24T06:21:57.046 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127728.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.127728.log.gz 2023-11-24T06:21:57.046 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127452.log 2023-11-24T06:21:57.046 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114975.log 2023-11-24T06:21:57.046 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114079.log 2023-11-24T06:21:57.046 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114975.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127452.log.gz 2023-11-24T06:21:57.046 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114975.log.gz 2023-11-24T06:21:57.046 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114079.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114079.log.gz 2023-11-24T06:21:57.047 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114125.log 2023-11-24T06:21:57.047 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117351.log 2023-11-24T06:21:57.047 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114883.log 2023-11-24T06:21:57.047 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2%/var/log/ceph/ceph-client.admin.117351.log: -- replaced with /var/log/ceph/ceph-client.admin.114125.log.gz 2023-11-24T06:21:57.047 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.117351.log.gz 2023-11-24T06:21:57.047 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114883.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114883.log.gz 2023-11-24T06:21:57.047 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117585.log 2023-11-24T06:21:57.048 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119473.log 2023-11-24T06:21:57.048 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120986.log 2023-11-24T06:21:57.048 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119473.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117585.log.gz 2023-11-24T06:21:57.048 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119473.log.gz 2023-11-24T06:21:57.048 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120986.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.120986.log.gz 2023-11-24T06:21:57.048 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110072.log 2023-11-24T06:21:57.048 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126566.log 2023-11-24T06:21:57.049 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.118331.log 2023-11-24T06:21:57.049 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126566.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110072.log.gz 2023-11-24T06:21:57.049 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126566.log.gz 2023-11-24T06:21:57.049 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.118331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.118331.log.gz 2023-11-24T06:21:57.049 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115148.log 2023-11-24T06:21:57.049 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124912.log 2023-11-24T06:21:57.049 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120932.log 2023-11-24T06:21:57.050 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124912.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115148.log.gz 2023-11-24T06:21:57.050 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124912.log.gz 2023-11-24T06:21:57.050 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120932.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120932.log.gz 2023-11-24T06:21:57.050 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114217.log 2023-11-24T06:21:57.050 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118823.log 2023-11-24T06:21:57.050 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123058.log 2023-11-24T06:21:57.050 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114217.log.gz 2023-11-24T06:21:57.050 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120455.log 2023-11-24T06:21:57.051 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123058.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118823.log.gz 2023-11-24T06:21:57.051 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.123058.log.gz 2023-11-24T06:21:57.051 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120455.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120455.log.gz 2023-11-24T06:21:57.051 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126817.log 2023-11-24T06:21:57.051 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126133.log 2023-11-24T06:21:57.051 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127682.log 2023-11-24T06:21:57.051 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.126817.log.gz 2023-11-24T06:21:57.052 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123196.log 2023-11-24T06:21:57.052 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127682.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126133.log.gz 2023-11-24T06:21:57.052 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123196.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.127682.log.gz 2023-11-24T06:21:57.052 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123104.log 2023-11-24T06:21:57.052 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123196.log.gz 2023-11-24T06:21:57.052 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-11-24T06:21:57.052 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124217.log 2023-11-24T06:21:57.053 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.123104.log.gz 2023-11-24T06:21:57.053 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115194.log 2023-11-24T06:21:57.053 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.124217.log: -- /var/log/ceph/ceph-client.admin.116669.log 2023-11-24T06:21:57.053 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115194.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.124217.log.gz 2023-11-24T06:21:57.053 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116669.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.115194.log.gz 2023-11-24T06:21:57.053 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117445.log 2023-11-24T06:21:57.053 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116669.log.gz 2023-11-24T06:21:57.054 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111467.log 2023-11-24T06:21:57.054 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126317.log 2023-11-24T06:21:57.054 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111467.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117445.log.gz 2023-11-24T06:21:57.054 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111467.log.gz 2023-11-24T06:21:57.054 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126317.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126317.log.gz 2023-11-24T06:21:57.054 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116301.log 2023-11-24T06:21:57.054 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118683.log 2023-11-24T06:21:57.055 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116301.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116301.log.gz 2023-11-24T06:21:57.055 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118683.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118683.log.gz 2023-11-24T06:21:57.055 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122229.log 2023-11-24T06:21:57.055 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127360.log 2023-11-24T06:21:57.055 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122229.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.122229.log.gz 2023-11-24T06:21:57.055 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127360.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.127360.log.gz 2023-11-24T06:21:57.055 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119933.log 2023-11-24T06:21:57.055 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125999.log 2023-11-24T06:21:57.056 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119933.log: /var/log/ceph/ceph-client.admin.125999.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.119933.log.gz 2023-11-24T06:21:57.056 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.125999.log.gz 2023-11-24T06:21:57.056 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121448.log 2023-11-24T06:21:57.056 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117028.log 2023-11-24T06:21:57.056 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121448.log: /var/log/ceph/ceph-client.admin.117028.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121448.log.gz 2023-11-24T06:21:57.056 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117028.log.gz 2023-11-24T06:21:57.056 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124636.log 2023-11-24T06:21:57.057 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118952.log 2023-11-24T06:21:57.057 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124636.log: /var/log/ceph/ceph-client.admin.118952.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124636.log.gz 2023-11-24T06:21:57.057 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.118952.log.gz 2023-11-24T06:21:57.057 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112372.log 2023-11-24T06:21:57.057 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116071.log 2023-11-24T06:21:57.057 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112372.log: /var/log/ceph/ceph-client.admin.116071.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.112372.log.gz 2023-11-24T06:21:57.057 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116071.log.gz 2023-11-24T06:21:57.058 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124774.log 2023-11-24T06:21:57.058 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127544.log 2023-11-24T06:21:57.058 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124774.log: /var/log/ceph/ceph-client.admin.127544.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.124774.log.gz 2023-11-24T06:21:57.058 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.127544.log.gz 2023-11-24T06:21:57.058 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126955.log 2023-11-24T06:21:57.058 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126520.log 2023-11-24T06:21:57.058 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126955.log: /var/log/ceph/ceph-client.admin.126520.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126955.log.gz 2023-11-24T06:21:57.058 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126520.log.gz 2023-11-24T06:21:57.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119611.log 2023-11-24T06:21:57.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113238.log 2023-11-24T06:21:57.059 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119611.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119611.log.gz 2023-11-24T06:21:57.059 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.113238.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.113238.log.gz 2023-11-24T06:21:57.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119044.log 2023-11-24T06:21:57.059 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124958.log 2023-11-24T06:21:57.059 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119044.log: /var/log/ceph/ceph-client.admin.124958.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.119044.log.gz 2023-11-24T06:21:57.060 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109717.log 2023-11-24T06:21:57.060 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124958.log.gz 2023-11-24T06:21:57.060 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120209.log 2023-11-24T06:21:57.061 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.103582.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.103582.log.gz 2023-11-24T06:21:57.061 INFO:teuthology.orchestra.run.smithi123.stderr: 95.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-11-24T06:21:57.078 INFO:teuthology.orchestra.run.smithi123.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-11-24T06:21:57.097 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109717.log: /var/log/ceph/ceph-client.admin.120209.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109717.log.gz 2023-11-24T06:21:57.097 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.120209.log.gz 2023-11-24T06:21:57.097 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124498.log 2023-11-24T06:21:57.098 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115286.log 2023-11-24T06:21:57.098 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124498.log: /var/log/ceph/ceph-client.admin.115286.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.124498.log.gz 2023-11-24T06:21:57.098 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115286.log.gz 2023-11-24T06:21:57.098 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121172.log 2023-11-24T06:21:57.098 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114632.log 2023-11-24T06:21:57.098 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121172.log: /var/log/ceph/ceph-client.admin.114632.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121172.log.gz 2023-11-24T06:21:57.098 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114632.log.gz 2023-11-24T06:21:57.099 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118487.log 2023-11-24T06:21:57.099 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118579.log 2023-11-24T06:21:57.099 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117744.log 2023-11-24T06:21:57.099 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118487.log.gz 2023-11-24T06:21:57.099 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118579.log: /var/log/ceph/ceph-client.admin.117744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111107.log 2023-11-24T06:21:57.099 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.118579.log.gz 2023-11-24T06:21:57.099 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116982.log 2023-11-24T06:21:57.100 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.117744.log.gz 2023-11-24T06:21:57.100 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103654.log 2023-11-24T06:21:57.100 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111107.log.gz 2023-11-24T06:21:57.100 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111866.log 2023-11-24T06:21:57.100 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116982.log.gz 2023-11-24T06:21:57.100 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.103654.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.103654.log.gz 2023-11-24T06:21:57.100 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111866.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111866.log.gz 2023-11-24T06:21:57.100 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109539.log 2023-11-24T06:21:57.101 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-24T06:21:57.101 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123491.log 2023-11-24T06:21:57.101 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109539.log.gz 2023-11-24T06:21:57.101 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.123491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117882.log 2023-11-24T06:21:57.101 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123491.log.gz 2023-11-24T06:21:57.101 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117882.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117882.log.gz 2023-11-24T06:21:57.101 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122966.log 2023-11-24T06:21:57.102 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111101.log 2023-11-24T06:21:57.102 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122966.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-24T06:21:57.102 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111101.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.122966.log.gz 2023-11-24T06:21:57.102 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111101.log.gz 2023-11-24T06:21:57.102 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111104.log 2023-11-24T06:21:57.102 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111470.log 2023-11-24T06:21:57.102 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119657.log 2023-11-24T06:21:57.102 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111104.log.gz 2023-11-24T06:21:57.103 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111470.log: /var/log/ceph/ceph-client.admin.119657.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111470.log.gz 2023-11-24T06:21:57.103 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.119657.log.gz 2023-11-24T06:21:57.103 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118533.log 2023-11-24T06:21:57.103 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127912.log 2023-11-24T06:21:57.103 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120347.log 2023-11-24T06:21:57.103 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127912.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.118533.log.gz 2023-11-24T06:21:57.103 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.127912.log.gz 2023-11-24T06:21:57.104 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120347.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120347.log.gz 2023-11-24T06:21:57.104 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114402.log 2023-11-24T06:21:57.104 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110018.log 2023-11-24T06:21:57.104 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114402.log: /var/log/ceph/ceph-client.admin.110018.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114402.log.gz 2023-11-24T06:21:57.104 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120651.log 2023-11-24T06:21:57.104 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110018.log.gz 2023-11-24T06:21:57.104 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122091.log 2023-11-24T06:21:57.104 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120651.log: /var/log/ceph/ceph-client.admin.122091.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.120651.log.gz 2023-11-24T06:21:57.105 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122091.log.gz 2023-11-24T06:21:57.105 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117121.log 2023-11-24T06:21:57.105 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.113008.log 2023-11-24T06:21:57.105 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117121.log: /var/log/ceph/ceph-client.0.113008.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117121.log.gz 2023-11-24T06:21:57.105 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.113008.log.gz 2023-11-24T06:21:57.105 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117213.log 2023-11-24T06:21:57.105 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114745.log 2023-11-24T06:21:57.106 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111086.log 2023-11-24T06:21:57.106 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2%/var/log/ceph/ceph-client.admin.114745.log: -- replaced with /var/log/ceph/ceph-client.admin.117213.log.gz 2023-11-24T06:21:57.106 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111086.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114745.log.gz 2023-11-24T06:21:57.106 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111086.log.gz 2023-11-24T06:21:57.106 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128004.log 2023-11-24T06:21:57.106 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120071.log 2023-11-24T06:21:57.106 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.128004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115466.log 2023-11-24T06:21:57.106 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.128004.log.gz 2023-11-24T06:21:57.107 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120071.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.120071.log.gz 2023-11-24T06:21:57.107 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112234.log 2023-11-24T06:21:57.107 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121218.log 2023-11-24T06:21:57.107 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7%/var/log/ceph/ceph-client.admin.112234.log: -- replaced with /var/log/ceph/ceph-client.admin.115466.log.gz 2023-11-24T06:21:57.107 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112234.log.gz 2023-11-24T06:21:57.107 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118028.log 2023-11-24T06:21:57.107 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126633.log 2023-11-24T06:21:57.108 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.121218.log.gz 2023-11-24T06:21:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118028.log: /var/log/ceph/ceph-client.admin.126633.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.118028.log.gz 2023-11-24T06:21:57.108 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.126633.log.gz 2023-11-24T06:21:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121494.log 2023-11-24T06:21:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.121939.log 2023-11-24T06:21:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123537.log 2023-11-24T06:21:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.121939.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.121494.log.gz 2023-11-24T06:21:57.109 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.121939.log.gz 2023-11-24T06:21:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121310.log 2023-11-24T06:21:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111820.log 2023-11-24T06:21:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121310.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.123537.log.gz 2023-11-24T06:21:57.109 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.121310.log.gz 2023-11-24T06:21:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114586.log 2023-11-24T06:21:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123750.log 2023-11-24T06:21:57.110 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114586.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111820.log.gz 2023-11-24T06:21:57.110 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.114586.log.gz 2023-11-24T06:21:57.110 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118998.log 2023-11-24T06:21:57.110 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127774.log 2023-11-24T06:21:57.110 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123750.log.gz 2023-11-24T06:21:57.110 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118998.log: 93.0% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-11-24T06:21:57.111 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127774.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118998.log.gz 2023-11-24T06:21:57.111 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128400.log 2023-11-24T06:21:57.111 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127774.log.gz 2023-11-24T06:21:57.111 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117790.log 2023-11-24T06:21:57.111 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.128400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127093.log 2023-11-24T06:21:57.111 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117790.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128400.log.gz 2023-11-24T06:21:57.111 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127093.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117790.log.gz 2023-11-24T06:21:57.111 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122397.log 2023-11-24T06:21:57.112 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127093.log.gz 2023-11-24T06:21:57.112 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122397.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122397.log.gz 2023-11-24T06:21:57.112 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128096.log 2023-11-24T06:21:57.112 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110734.log 2023-11-24T06:21:57.112 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.128096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110486.log 2023-11-24T06:21:57.112 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128096.log.gz 2023-11-24T06:21:57.112 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122289.log 2023-11-24T06:21:57.113 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122489.log 2023-11-24T06:21:57.113 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110486.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110734.log.gz 2023-11-24T06:21:57.113 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119316.log 2023-11-24T06:21:57.113 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110486.log.gz 2023-11-24T06:21:57.113 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122489.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.122289.log.gz 2023-11-24T06:21:57.113 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.122489.log.gz 2023-11-24T06:21:57.113 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119316.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.119316.log.gz 2023-11-24T06:21:57.113 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114171.log 2023-11-24T06:21:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.118335.log 2023-11-24T06:21:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127590.log 2023-11-24T06:21:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.118335.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114171.log.gz 2023-11-24T06:21:57.114 INFO:teuthology.orchestra.run.smithi089.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.118335.log.gz 2023-11-24T06:21:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127590.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.127590.log.gz 2023-11-24T06:21:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.125403.log 2023-11-24T06:21:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.125399.log 2023-11-24T06:21:57.115 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.125403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.125403.log.gz 2023-11-24T06:21:57.115 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123650.log 2023-11-24T06:21:57.115 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123604.log 2023-11-24T06:21:57.115 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.125399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.125399.log.gz 2023-11-24T06:21:57.115 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126428.log 2023-11-24T06:21:57.115 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123650.log.gz 2023-11-24T06:21:57.115 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110348.log 2023-11-24T06:21:57.115 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114929.log 2023-11-24T06:21:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126428.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123604.log.gz 2023-11-24T06:21:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110348.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-24T06:21:57.116 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126428.log.gz 2023-11-24T06:21:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114929.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110348.log.gz 2023-11-24T06:21:57.116 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.114929.log.gz 2023-11-24T06:21:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123012.log 2023-11-24T06:21:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110736.log 2023-11-24T06:21:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118098.log 2023-11-24T06:21:57.117 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.123012.log.gz 2023-11-24T06:21:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110736.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110736.log.gz 2023-11-24T06:21:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113987.log 2023-11-24T06:21:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119703.log 2023-11-24T06:21:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.113987.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118098.log.gz 2023-11-24T06:21:57.117 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113987.log.gz 2023-11-24T06:21:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116347.log 2023-11-24T06:21:57.118 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103213.log 2023-11-24T06:21:57.118 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116347.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119703.log.gz 2023-11-24T06:21:57.118 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116347.log.gz 2023-11-24T06:21:57.118 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.103213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115512.log 2023-11-24T06:21:57.118 INFO:teuthology.orchestra.run.smithi089.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.103213.log.gz 2023-11-24T06:21:57.118 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117075.log 2023-11-24T06:21:57.118 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122855.log 2023-11-24T06:21:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117075.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.115512.log.gz 2023-11-24T06:21:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122855.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117075.log.gz 2023-11-24T06:21:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120790.log 2023-11-24T06:21:57.119 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122855.log.gz 2023-11-24T06:21:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114837.log 2023-11-24T06:21:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120790.log: /var/log/ceph/ceph-client.admin.114837.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.120790.log.gz 2023-11-24T06:21:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119565.log 2023-11-24T06:21:57.119 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.114837.log.gz 2023-11-24T06:21:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120743.log 2023-11-24T06:21:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.113673.log 2023-11-24T06:21:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120743.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.119565.log.gz 2023-11-24T06:21:57.120 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120743.log.gz 2023-11-24T06:21:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.2.113673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.113673.log.gz 2023-11-24T06:21:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111466.log 2023-11-24T06:21:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122671.log 2023-11-24T06:21:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111466.log: /var/log/ceph/ceph-client.admin.122671.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111466.log.gz 2023-11-24T06:21:57.121 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.122671.log.gz 2023-11-24T06:21:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119362.log 2023-11-24T06:21:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124728.log 2023-11-24T06:21:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119362.log: /var/log/ceph/ceph-client.admin.124728.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119362.log.gz 2023-11-24T06:21:57.121 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.124728.log.gz 2023-11-24T06:21:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113895.log 2023-11-24T06:21:57.122 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125753.log 2023-11-24T06:21:57.122 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.113895.log: /var/log/ceph/ceph-client.admin.125753.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113895.log.gz 2023-11-24T06:21:57.122 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.125753.log.gz 2023-11-24T06:21:57.122 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121356.log 2023-11-24T06:21:57.122 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115707.log 2023-11-24T06:21:57.122 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121356.log: /var/log/ceph/ceph-client.admin.115707.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.121356.log.gz 2023-11-24T06:21:57.122 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.115707.log.gz 2023-11-24T06:21:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117493.log 2023-11-24T06:21:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123380.log 2023-11-24T06:21:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117493.log: /var/log/ceph/ceph-client.admin.123380.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117493.log.gz 2023-11-24T06:21:57.123 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123380.log.gz 2023-11-24T06:21:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115753.log 2023-11-24T06:21:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123842.log 2023-11-24T06:21:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115753.log: /var/log/ceph/ceph-client.admin.123842.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.115753.log.gz 2023-11-24T06:21:57.123 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.123842.log.gz 2023-11-24T06:21:57.124 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119090.log 2023-11-24T06:21:57.124 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116715.log 2023-11-24T06:21:57.124 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115799.log 2023-11-24T06:21:57.124 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116715.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.119090.log.gz 2023-11-24T06:21:57.124 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115799.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116715.log.gz 2023-11-24T06:21:57.124 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117167.log 2023-11-24T06:21:57.124 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115799.log.gz 2023-11-24T06:21:57.125 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111682.log 2023-11-24T06:21:57.125 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117167.log: /var/log/ceph/ceph-client.admin.111682.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.117167.log.gz 2023-11-24T06:21:57.125 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111682.log.gz 2023-11-24T06:21:57.125 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125907.log 2023-11-24T06:21:57.125 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110738.log 2023-11-24T06:21:57.125 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125907.log: /var/log/ceph/ceph-client.admin.110738.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.125907.log.gz 2023-11-24T06:21:57.125 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110738.log.gz 2023-11-24T06:21:57.125 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116439.log 2023-11-24T06:21:57.126 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110739.log 2023-11-24T06:21:57.126 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114448.log 2023-11-24T06:21:57.126 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110739.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116439.log.gz 2023-11-24T06:21:57.126 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114448.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110739.log.gz 2023-11-24T06:21:57.126 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114448.log.gz 2023-11-24T06:21:57.126 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111728.log 2023-11-24T06:21:57.126 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118441.log 2023-11-24T06:21:57.127 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111728.log: /var/log/ceph/ceph-client.admin.118441.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.111728.log.gz 2023-11-24T06:21:57.127 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118441.log.gz 2023-11-24T06:21:57.127 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112526.log 2023-11-24T06:21:57.127 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124406.log 2023-11-24T06:21:57.127 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111106.log 2023-11-24T06:21:57.127 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.124406.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112526.log.gz 2023-11-24T06:21:57.127 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111106.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124406.log.gz 2023-11-24T06:21:57.128 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111106.log.gz 2023-11-24T06:21:57.128 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128310.log 2023-11-24T06:21:57.128 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124040.log 2023-11-24T06:21:57.128 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.128310.log: /var/log/ceph/ceph-client.admin.124040.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.128310.log.gz 2023-11-24T06:21:57.128 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124040.log.gz 2023-11-24T06:21:57.128 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116577.log 2023-11-24T06:21:57.128 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117982.log 2023-11-24T06:21:57.128 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.116577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128188.log 2023-11-24T06:21:57.129 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117982.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116577.log.gz 2023-11-24T06:21:57.129 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117982.log.gz 2023-11-24T06:21:57.129 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.128188.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128188.log.gz 2023-11-24T06:21:57.129 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112004.log 2023-11-24T06:21:57.129 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115240.log 2023-11-24T06:21:57.129 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112004.log: /var/log/ceph/ceph-client.admin.115240.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112004.log.gz 2023-11-24T06:21:57.129 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115240.log.gz 2023-11-24T06:21:57.130 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125004.log 2023-11-24T06:21:57.130 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110730.log 2023-11-24T06:21:57.130 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.125004.log: /var/log/ceph/ceph-client.admin.110730.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.125004.log.gz 2023-11-24T06:21:57.130 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110730.log.gz 2023-11-24T06:21:57.130 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109670.log 2023-11-24T06:21:57.130 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115845.log 2023-11-24T06:21:57.130 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109670.log: /var/log/ceph/ceph-client.admin.115845.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109670.log.gz 2023-11-24T06:21:57.130 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.115845.log.gz 2023-11-24T06:21:57.131 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115933.log 2023-11-24T06:21:57.131 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122763.log 2023-11-24T06:21:57.131 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.115933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120884.log 2023-11-24T06:21:57.131 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.122763.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.115933.log.gz 2023-11-24T06:21:57.131 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.120884.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122763.log.gz 2023-11-24T06:21:57.131 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.120884.log.gz 2023-11-24T06:21:57.131 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127001.log 2023-11-24T06:21:57.132 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118625.log 2023-11-24T06:21:57.132 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127001.log: /var/log/ceph/ceph-client.admin.118625.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.127001.log.gz 2023-11-24T06:21:57.132 INFO:teuthology.orchestra.run.smithi089.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.118625.log.gz 2023-11-24T06:21:57.132 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110256.log 2023-11-24T06:21:57.132 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121702.log 2023-11-24T06:21:57.132 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110256.log: /var/log/ceph/ceph-client.admin.121702.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110256.log.gz 2023-11-24T06:21:57.132 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.121702.log.gz 2023-11-24T06:21:57.132 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103523.log 2023-11-24T06:21:57.133 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111774.log 2023-11-24T06:21:57.133 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.103523.log: /var/log/ceph/ceph-client.admin.111774.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103523.log.gz 2023-11-24T06:21:57.133 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.111774.log.gz 2023-11-24T06:21:57.133 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126474.log 2023-11-24T06:21:57.133 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116851.log 2023-11-24T06:21:57.133 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126474.log: /var/log/ceph/ceph-client.admin.116851.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126474.log.gz 2023-11-24T06:21:57.133 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116851.log.gz 2023-11-24T06:21:57.134 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117539.log 2023-11-24T06:21:57.134 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121540.log 2023-11-24T06:21:57.134 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114356.log 2023-11-24T06:21:57.134 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121540.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.117539.log.gz 2023-11-24T06:21:57.134 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114356.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121540.log.gz 2023-11-24T06:21:57.134 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.114356.log.gz 2023-11-24T06:21:57.134 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127958.log 2023-11-24T06:21:57.134 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114791.log 2023-11-24T06:21:57.135 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127958.log: /var/log/ceph/ceph-client.admin.114791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119519.log 2023-11-24T06:21:57.135 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127958.log.gz 2023-11-24T06:21:57.135 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114791.log.gz 2023-11-24T06:21:57.135 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124125.log 2023-11-24T06:21:57.135 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119519.log: /var/log/ceph/ceph-client.admin.124125.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119519.log.gz 2023-11-24T06:21:57.135 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112280.log 2023-11-24T06:21:57.135 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124125.log.gz 2023-11-24T06:21:57.136 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110210.log 2023-11-24T06:21:57.136 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.112280.log: /var/log/ceph/ceph-client.admin.110210.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112280.log.gz 2023-11-24T06:21:57.136 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110210.log.gz 2023-11-24T06:21:57.136 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128142.log 2023-11-24T06:21:57.136 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111958.log 2023-11-24T06:21:57.136 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.128142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128234.log 2023-11-24T06:21:57.136 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111958.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.128142.log.gz 2023-11-24T06:21:57.136 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.128234.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111958.log.gz 2023-11-24T06:21:57.137 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.128234.log.gz 2023-11-24T06:21:57.137 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126382.log 2023-11-24T06:21:57.138 INFO:teuthology.orchestra.run.smithi123.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-24T06:21:57.138 INFO:teuthology.orchestra.run.smithi123.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-24T06:21:57.144 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119136.log 2023-11-24T06:21:57.144 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126382.log: /var/log/ceph/ceph-client.admin.119136.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.126382.log.gz 2023-11-24T06:21:57.144 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119136.log.gz 2023-11-24T06:21:57.144 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109624.log 2023-11-24T06:21:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123334.log 2023-11-24T06:21:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126045.log 2023-11-24T06:21:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.123334.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.109624.log.gz 2023-11-24T06:21:57.145 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123334.log.gz 2023-11-24T06:21:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126045.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.126045.log.gz 2023-11-24T06:21:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126909.log 2023-11-24T06:21:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-24T06:21:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.126909.log: /var/log/ceph/ceph-mon.a.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126909.log.gz 2023-11-24T06:21:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117677.log 2023-11-24T06:21:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122443.log 2023-11-24T06:21:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.117677.log: /var/log/ceph/ceph-client.admin.122443.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.117677.log.gz 2023-11-24T06:21:57.146 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122443.log.gz 2023-11-24T06:21:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114540.log 2023-11-24T06:21:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116936.log 2023-11-24T06:21:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.114540.log: /var/log/ceph/ceph-client.admin.116936.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114540.log.gz 2023-11-24T06:21:57.147 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116936.log.gz 2023-11-24T06:21:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111912.log 2023-11-24T06:21:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125166.log 2023-11-24T06:21:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.111912.log: /var/log/ceph/ceph-client.admin.125166.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111912.log.gz 2023-11-24T06:21:57.147 INFO:teuthology.orchestra.run.smithi089.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.125166.log.gz 2023-11-24T06:21:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109334.log 2023-11-24T06:21:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115661.log 2023-11-24T06:21:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.109334.log: /var/log/ceph/ceph-client.admin.115661.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109334.log.gz 2023-11-24T06:21:57.148 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115661.log.gz 2023-11-24T06:21:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121078.log 2023-11-24T06:21:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-24T06:21:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.121078.log: /var/log/ceph/ceph-mgr.y.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.121078.log.gz 2023-11-24T06:21:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118775.log 2023-11-24T06:21:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.118775.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118775.log.gz 2023-11-24T06:21:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127820.log 2023-11-24T06:21:57.149 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.127820.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.127820.log.gz 2023-11-24T06:21:57.149 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119841.log 2023-11-24T06:21:57.149 INFO:teuthology.orchestra.run.smithi089.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-24T06:21:57.149 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122183.log 2023-11-24T06:21:57.149 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119841.log: /var/log/ceph/ceph-client.admin.122183.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119841.log.gz 2023-11-24T06:21:57.149 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110731.log 2023-11-24T06:21:57.149 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122183.log.gz 2023-11-24T06:21:57.149 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122717.log 2023-11-24T06:21:57.150 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.110731.log: /var/log/ceph/ceph-client.admin.122717.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110731.log.gz 2023-11-24T06:21:57.150 INFO:teuthology.orchestra.run.smithi089.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.122717.log.gz 2023-11-24T06:21:57.150 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119270.log 2023-11-24T06:21:57.150 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120697.log 2023-11-24T06:21:57.150 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.119270.log: /var/log/ceph/ceph-client.admin.120697.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119270.log.gz 2023-11-24T06:21:57.150 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.120697.log.gz 2023-11-24T06:21:57.156 INFO:teuthology.orchestra.run.smithi123.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-24T06:21:57.158 INFO:teuthology.orchestra.run.smithi123.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-24T06:21:57.169 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-24T06:21:57.181 INFO:teuthology.orchestra.run.smithi089.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-11-24T06:21:57.183 INFO:teuthology.orchestra.run.smithi123.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-24T06:21:57.198 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-24T06:21:57.226 INFO:teuthology.orchestra.run.smithi123.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-24T06:21:57.229 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-24T06:21:57.229 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m0.273s 2023-11-24T06:21:57.229 INFO:teuthology.orchestra.run.smithi123.stderr:user 0m1.255s 2023-11-24T06:21:57.230 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m0.062s 2023-11-24T06:21:57.252 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-24T06:21:57.367 INFO:teuthology.orchestra.run.smithi089.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-24T06:21:57.369 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-11-24T06:21:57.369 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m0.439s 2023-11-24T06:21:57.369 INFO:teuthology.orchestra.run.smithi089.stderr:user 0m1.315s 2023-11-24T06:21:57.369 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m0.210s 2023-11-24T06:21:57.369 INFO:tasks.ceph:Archiving logs... 2023-11-24T06:21:57.370 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/remote/smithi089/log 2023-11-24T06:21:57.371 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-24T06:21:57.889 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/remote/smithi123/log 2023-11-24T06:21:57.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-24T06:21:58.103 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-24T06:21:58.123 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-24T06:21:58.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-24T06:21:58.126 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-24T06:21:58.209 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-24T06:21:58.219 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-24T06:21:58.219 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T06:21:58.222 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T06:21:58.236 INFO:teuthology.orchestra.run.smithi089.stderr:bash: ntpq: command not found 2023-11-24T06:21:58.239 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2023-11-24T06:21:58.240 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T06:21:58.240 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2023-11-24T06:21:58.241 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 60 +1741us[+1743us] +/- 45ms 2023-11-24T06:21:58.241 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -2082us[-2080us] +/- 65ms 2023-11-24T06:21:58.241 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -423us[ -421us] +/- 97ms 2023-11-24T06:21:58.241 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:21:58.243 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T06:21:58.243 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2023-11-24T06:21:58.245 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 61 +1750us[+1763us] +/- 45ms 2023-11-24T06:21:58.245 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -2078us[-2065us] +/- 65ms 2023-11-24T06:21:58.245 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -421us[ -408us] +/- 97ms 2023-11-24T06:21:58.245 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:21:58.246 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-24T06:21:58.256 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-24T06:21:58.257 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-24T06:21:58.271 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-24T06:21:58.386 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-24T06:21:58.481 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T06:21:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:21:58.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T06:21:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:21:58.546 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-24T06:21:58.557 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-24T06:21:58.573 INFO:teuthology.task.internal:Duration was 913.512664 seconds 2023-11-24T06:21:58.573 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-24T06:21:58.588 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-24T06:21:58.588 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-24T06:21:58.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-24T06:21:58.635 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T06:21:58.637 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T06:21:58.799 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-24T06:21:58.800 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2023-11-24T06:21:58.800 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-24T06:21:58.822 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2023-11-24T06:21:58.822 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-24T06:21:58.845 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-24T06:21:58.846 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-24T06:21:58.864 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-24T06:21:58.925 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-24T06:21:58.925 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-24T06:21:58.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-24T06:21:59.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-24T06:21:59.051 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-24T06:21:59.051 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-24T06:21:59.077 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-24T06:21:59.104 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-24T06:21:59.115 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-24T06:21:59.120 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-24T06:21:59.143 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2023-11-24T06:21:59.170 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2023-11-24T06:21:59.185 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-24T06:21:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:21:59.213 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-24T06:21:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:21:59.241 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-24T06:21:59.251 INFO:teuthology.task.internal:Transferring archived files... 2023-11-24T06:21:59.252 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/remote/smithi089 2023-11-24T06:21:59.252 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-24T06:21:59.300 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556/remote/smithi123 2023-11-24T06:21:59.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-24T06:21:59.338 INFO:teuthology.task.internal:Removing archive directory... 2023-11-24T06:21:59.338 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-24T06:21:59.340 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-24T06:21:59.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-24T06:21:59.401 INFO:teuthology.task.internal:Not uploading archives. 2023-11-24T06:21:59.401 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-24T06:21:59.422 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-24T06:21:59.423 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-24T06:21:59.425 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-24T06:21:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 24 06:21 /home/ubuntu/cephtest 2023-11-24T06:21:59.446 INFO:teuthology.orchestra.run.smithi123.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 24 06:21 /home/ubuntu/cephtest 2023-11-24T06:21:59.447 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-24T06:21:59.458 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-24T06:21:59.498 INFO:teuthology.nuke:Checking targets against current locks 2023-11-24T06:21:59.519 DEBUG:teuthology.nuke:shortname: smithi089 2023-11-24T06:21:59.520 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T06:21:59.541 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 06:01:31.131005', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-24T06:21:59.546 DEBUG:teuthology.nuke:shortname: smithi123 2023-11-24T06:21:59.546 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T06:21:59.564 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 06:01:31.132501', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-24T06:21:59.592 INFO:teuthology.orchestra.console.smithi089:Power off 2023-11-24T06:21:59.593 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-24T06:21:59.609 INFO:teuthology.orchestra.console.smithi123:Power off 2023-11-24T06:21:59.609 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-24T06:21:59.618 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2023-11-24T06:21:59.618 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:21:59.630 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2023-11-24T06:21:59.630 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:21:59.642 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-11-24T06:21:59.655 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2023-11-24T06:22:03.643 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:22:03.655 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:22:03.755 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-11-24T06:22:03.766 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2023-11-24T06:22:07.758 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:22:07.767 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:22:07.869 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-11-24T06:22:07.877 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2023-11-24T06:22:07.977 INFO:teuthology.orchestra.console.smithi123:Power off completed 2023-11-24T06:22:11.872 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:22:11.981 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2023-11-24T06:22:12.082 INFO:teuthology.orchestra.console.smithi089:Power off completed 2023-11-24T06:22:12.187 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} duration: 913.512663602829 failure_reason: 'Test failure: test_volume_info_pending_subvol_deletions (tasks.cephfs.test_volumes.TestVolumes)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0a6aa6a38f4a4a13bf3d798a7aa30b47 status: fail success: false 2023-11-24T06:22:12.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-24T06:22:12.268 INFO:teuthology.run:FAIL